Control system with task manager

ABSTRACT

A system includes a first robotic machine having a first set of capabilities for interacting with a target object; a second robotic machine having a second set of capabilities for interacting with the target object; and a task manager that can determine capability requirements to perform a task on the target object. The task has an associated series of sub-tasks. The task manager can assign a first sequence of sub-tasks for performance by the first robotic machine based on the first set of capabilities and a second sequence of sub-tasks for performance by the second robotic machine based on the second set of capabilities. The first and second robotic machines can coordinate performance of the first sequence of sub-tasks by the first robotic machine with performance of the second sequence of sub-tasks by the second robotic machine to accomplish the task.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/411,788, filed 14 May 2019, which is a continuation-in-partof U.S. patent application Ser. No. 16/379,976, filed 10 Apr. 2019,which is a continuation of U.S. patent application Ser. No. 16/114,318(“the '318 Application”), filed on 28 Aug. 2018.

The '318 Application is a continuation-in-part of patented U.S.application Ser. No. 15/198,673, filed on Jun. 30, 2016 and published asU.S. Pat. No. 10,065,317; and is a continuation-in-part of pending U.S.application Ser. No. 15/399,313, filed on Jan. 5, 2017; and is acontinuation-in-part of patented U.S. application Ser. No. 15/183,850,filed on Jun. 16, 2016 and published as U.S. Pat. No. 10,105,844; and isa continuation-in-part of pending U.S. application Ser. No. 15/872,582,filed on Jan. 16, 2018; and is a continuation-in-part of pending U.S.application Ser. No. 15/809,515, filed on Nov. 10, 2017; and is acontinuation-in-part of pending U.S. application Ser. No. 15/804,767,filed on Nov. 6, 2017; and is a continuation-in-part of pending U.S.application Ser. No. 15/585,502, filed on May 3, 2017; and is acontinuation-in-part of pending U.S. application Ser. No. 15/587,950,filed on May 5, 2017; and is a continuation-in-part of pending U.S.application Ser. No. 15/473,384, filed on Mar. 29, 2017; and is acontinuation-in-part of patented U.S. application Ser. No. 14/541,370,filed on Nov. 14, 2014 and published as U.S. Pat. No. 10,110,795; and isa continuation-in-part of pending U.S. application Ser. No. 15/584,995,filed on May 2, 2017; and is a continuation-in-part of pending U.S.application Ser. No. 15/473,345, filed on Mar. 29, 2017, which claimspriority to U.S. Provisional Application No. 62/343,615, filed on May31, 2016 and to U.S. Provisional Application No. 62/336,332, filed onMay 13, 2016.

The '318 Application is also a continuation-in-part of patented U.S.application Ser. No. 15/058,494 filed on Mar. 2, 2016 and published asU.S. Pat. No. 10,093,022, which claims priority to U.S. ProvisionalApplication Nos. 62/269,523, 62/269,425, 62/269,377, and 62/269,481, allof which were filed on Dec. 18, 2015.

All the applications above are herein incorporated by reference in theirentireties, including the drawings, for all purposes.

BACKGROUND Technical Field

The subject matter described herein relates to a system that includes aplurality of robotic machines for performing tasks.

Discussion of Art

In some situations, the use of human operators may sometimes beundesirable. But, automated systems may pose problems as well. It may bedesirable to have a system that differs from those systems that arecurrently available.

BRIEF DESCRIPTION

In an embodiment, a system includes a first robotic machine having afirst set of capabilities for interacting with a target object; a secondrobotic machine having a second set of capabilities for interacting withthe target object; and a task manager having one or more processors andthat can determine capability requirements to perform a task on thetarget object. The task has an associated series of sub-tasks, with thesub-tasks having one or more capability requirements. The task managercan assign a first sequence of sub-tasks to the first robotic machinefor performance by the first robotic machine based at least in part onthe first set of capabilities and a second sequence of sub-tasks to thesecond robotic machine for performance by the second robotic machinebased at least in part on the second set of capabilities. The first andsecond robotic machines can coordinate performance of the first sequenceof sub-tasks by the first robotic machine with performance of the secondsequence of sub-tasks by the second robotic machine, and thereby toaccomplish the task.

In an embodiment, a system includes a first robotic machine having afirst set of capabilities for interacting with a surrounding environmentand a second robotic machine having a second set of capabilities forinteracting with the surrounding environment. The first robotic machinecan receive a first sequence of sub-tasks related to the first set ofcapabilities of the first robotic machine. The second robotic machinecan receive a second sequence of sub-tasks related to the second set ofcapabilities of the second robotic machine. The first and second roboticmachines can perform the first and second sequences of sub-tasks,respectively, to accomplish a task that involves at least one ofmanipulating or inspecting a target object that is separate from thefirst and second robotic machines. The first and second robotic machinescan coordinate performance of the first sequence of sub-tasks by thefirst robotic machine with performance of the second sequence ofsub-tasks by the second robotic machine.

In an embodiment, a method is provided for a first robotic machinehaving a first set of capabilities for interacting with a surroundingenvironment, where the first robotic machine receives a first sequenceof sub-tasks related to the first set of capabilities of the firstrobotic machine, and a second robotic machine having a second set ofcapabilities for interacting with the surrounding environment, where thesecond robotic machine receives a second sequence of sub-tasks relatedto the second set of capabilities of the second robotic machine. Themethod includes performing the first and second sequences of sub-tasksto accomplish a task comprising at least one of manipulating orinspecting a target object. Performance of the first sequence ofsub-tasks by the first robotic machine is coordinated with performanceof the second sequence of sub-tasks by the second robotic machine.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter described herein may be understood from reading thefollowing description of non-limiting embodiments, with reference to theattached drawings, wherein below:

FIG. 1 schematically illustrates a vehicle system, a first roboticmachine, and a second robotic machine according to one embodiment;

FIG. 2 illustrates one embodiment of the first robotic machine shown inFIG. 1;

FIG. 3 is a schematic block diagram of a control system for controllingfirst and second robotic machines to collaborate to perform an assignedtask on a vehicle;

FIG. 4 is a flow diagram showing interactions of a task manager and thefirst and second robotic machines of FIG. 3 to control and coordinatethe performance of an assigned task by the robotic machines on a vehicleaccording to an embodiment;

FIG. 5 is a block flow diagram showing a first sequence of sub-tasksassigned to a first robotic machine and a second sequence of sub-tasksassigned to a second robotic machine for performance of an assigned taskon a vehicle according to an embodiment;

FIG. 6 is a perspective view of two robotic machines collaborating toperform an assigned task on a vehicle according to another embodiment;and

FIG. 7 is a perspective view of two robotic machines collaborating toperform an assigned task on a first vehicle according to yet anotherembodiment.

DETAILED DESCRIPTION

The systems and methods described herein can be used to perform anassigned task on equipment using multiple robotic machines thatcollaborate to accomplish the assigned task. In some embodiments, theequipment or target object used to demonstrate aspects of this inventioncan be a vehicle or stationary equipment. In one embodiment, theequipment may be characterized as infrastructure. The nature of theequipment may require specific configuration of the inventive system,but each system may be selected to address application specificparameters. These selected features may include sensor packages, sizeand scale, implements, mobility platforms for one or more the multipleautomated robotic machines, and the like. Further, the internalmechanisms of the robotic machines may be selected based on applicationspecific parameters. Suitable mechanisms may be selected with regard tothe range of torque, type of fuel or energy, environmental tolerances,and the like.

The assigned task may involve at least one of the robotic machineassemblies approaching, engaging, modifying, and manipulating (e.g.,moving) a target object on the equipment. For example, a first roboticmachine may perform at least a portion of the assigned task by graspinga lever and pulling a lever with a specific force (e.g., torque) in aspecific direction and for a specific distance, before releasing thelever or returning the lever to a starting position. A second roboticmachine may collaborate with the first robotic machine in theperformance of the assigned task by at least one of inspecting aposition of the brake lever on the equipment, carrying the first roboticmachine to the equipment, lifting the first robotic machine toward thelever, verifying that the assigned task has been successfully completed,or the like. Thus, the multiple robotic machines work together toperform the assigned task. Each robotic machine performs at least onesub-task, and the assigned task may be completed upon the roboticmachines completing the sub-tasks. In order to collaborate successfully,the robotic machines may communicate directly with each other.

The robotic machines may perform the same or similar tasks on multipleitems of equipment in a larger equipment system. The robotic machinesmay perform the same or similar tasks on different types of theequipment and/or on different the equipment systems. Although tworobotic machines may be described in the example above, more than tworobotic machines may collaborate with each other to perform an assignedtask in another embodiment. For example, one robotic machine may flyalong the equipment to inspect a position of a lever or valve, a secondrobotic machine may lift a third robotic machine to the lever, and thethird robotic machine may grasp and manipulate the lever to change thelever position. In one use case, an example of an assigned task may beto release air from a hydraulic or a compressed air system on theequipment. If the compressed air system is a vehicle air brake system,the task may be referred to herein as brake bleeding.

In one or more embodiments, multiple robotic machines may be controlledto work together (e.g., collaborate) to perform different tasks to theequipment. The robotic machines may be automated, such that the tasksmay be performed autonomously without direct, immediate control of therobotic machines by a human operator as the robotic machines operate.The multiple robotic machines that collaborate with each other toperform an assigned task may be not identical (e.g., like copies). Therobotic machines have different capabilities or affordances relative toeach other. The robotic machines may be controlled to collaborate witheach other to perform a given assigned task because the task cannot becompleted by one of the robotic machines acting alone and/or the taskcan be completed by one of the robotic machines acting alone but not ina timely or cost-effective manner relative to multiple robotic machinesacting together to accomplish the assigned task. For example, in a railyard, some tasks include brake bleeding, actuating (e.g., setting orreleasing) hand brakes on two adjacent rail vehicles, connecting airhoses between the two adjacent rail vehicles (referred to herein as hoselacing), and the like.

In one embodiment a system includes a first robotic machine having afirst set of capabilities for interacting with a target object. A secondrobotic machine has a second set of capabilities for interacting withthe target object. A task manager has one or more processors and candetermine capability requirements to perform a task on the targetobject. The task has an associated series of sub-tasks having one ormore capability requirements. The task manager may assign a firstsequence of sub-tasks to the first robotic machine for performance bythe first robotic machine based at least in part on the first set ofcapabilities and a second sequence of sub-tasks to the second roboticmachine for performance by the second robotic machine based at least inpart on the second set of capabilities. The first and second roboticmachines may coordinate performance of the first sequence of sub-tasksby the first robotic machine with performance of the second sequence ofsub-tasks by the second robotic machine to accomplish the task.

Suitable first and second sets of capabilities of the first and secondrobotic machines may include least one of flying, driving, diving,lifting, imaging, grasping, rotating, tilting, extending, retracting,pushing, and/or pulling. Suitable capabilities of the second set of thesecond robotic machine may include at least one capability that differsfrom the first set of capabilities of the first robotic machine. (Forexample, one or more of the second set of capabilities of the secondrobotic machine may be capabilities that the first robotic machinelacks, and/or one or more of the first set of capabilities of the firstrobotic machine may be capabilities that the second robotic machinelacks.)

During operation, the first and second robotic machines may coordinateperformance of the first sequence of sub-tasks by the first roboticmachine with the performance of the second sequence of sub-tasks by thesecond robotic machine by communicating directly with each other. Thefirst robotic machine may notify the second robotic machine, directly orindirectly, that the corresponding sub-task is completed and the secondrobotic machine responds to the notification by completing acorresponding sub-task in the second sequence. The first robotic machinemay provide to the second robotic machine, directly or indirectly, asensor signal having information about the target object, and the taskmanager makes a decision whether the second robotic machine proceedswith a sub-task of the second sequence based at least in part on thesensor signal. At least some of the sub-tasks may be sequential suchthat the second robotic machine begins performance of a dependentsub-task in the second sequence responsive to receiving a notificationfrom the first robotic machine that the first robotic machine hascompleted a specific sub-task in the first sequence. The first roboticmachine may perform at least one of the sub-tasks in the first sequenceconcurrently with performance of at least one of the sub-tasks in thesecond sequence by the second robotic machine.

The task manager may access a database that stores capabilitydescriptions corresponding to each of plural robotic machines in a groupof robotic machines. (For example, the group of robotic machines maycomprise robotic machines that are available for selective use in agiven facility or other location.) The task manager may select from thegroup the first and second robotic machines appropriate to perform thetask instead of other robotic machines in the group based on asuitability of the capability descriptions of the first and secondrobotic machines to the task or corresponding sub-task.

In one example, the first robotic machine performs one or more of thefirst sequence of sub-tasks by coupling to and lifting the secondrobotic machine from a starting location to a lifted location such thatthe second robotic machine in the lifted location is positioned relativeto the target object to complete one or more of the second sequence ofsub-tasks than when the second robotic machine is in the startinglocation. (For example, it may be the case that the second roboticmachine cannot complete the one or more of the second sequence of thesub-tasks when in the starting location.) In another embodiment, thefirst robotic machine performs the first sequence of sub-tasks byflying, and the first robotic machine identifies the target object anddetermines at least two of a position of the target object, a positionof the first robotic machine, and/or a position of the second roboticmachine. The second robotic machine performs the second sequence ofsub-tasks by one or more of modifying the target object, manipulatingthe target object, observing the target object, interacting with thetarget object, and/or releasing the target object.

The first robotic machine, having been assigned a sequence of sub-tasksby the task manager, may determine to travel a determined path from afirst location to a second location of the first robotic machine, andthen signals to the second robotic machine, to the task manager, or boththe second robotic machine and the task manager information includingthe determined path, the act of using the capability, or both.Additionally or alternatively, the first robotic machine may determineto act using a capability of the first set of capabilities, or bothdetermines to travel the intended path and determines to act using thecapability.

In one embodiment, the second robotic machine, responsive to the signalfrom the first robotic machine, initiates a confirmatory receipt signalback to the first robotic machine. This may act as a “ready” signal toinitiate a sub-task. Suitable sub-tasks may include moving the roboticmachine, moving an implement of the robotic machine, transferringinformation or data, evaluating a sensor signal, and the like.

The first robotic machine and the second robotic machine each maygenerate one or more of time indexing signals associated with one orboth of the first sequence of sub-tasks and/or the second sequence ofsub-tasks, position indexing signals for locations of one or both of thefirst robotic machine and/or the second robotic machine, and/ororientation indexing signals for one or more tools to implement one orboth of the first set of capabilities of the first robotic machine andthe second set of capabilities of the second robotic machine.

At least one of the first robotic machine and/or the second roboticmachine may have a plurality of moving operational modes. In oneembodiment, the machine may have a first mode of operation that is agross movement mode and a second mode of operation that is a finemovement mode. Suitable robotic machines may include one or more of oneor more of a stabilizer, an outrigger, or a clamp. In one embodiment,suitable modes may include a transition in operation from the first modeto the second mode that includes deploying and setting the stabilizer,outrigger, or clamp. In another embodiment, there may be a fast-closeoperation that moves one or more robotic machines proximate to thetarget object quickly, followed by a transition to a slower movementmode that carefully moves the robotic machine from proximate the targetobject to contact with the target object. (A slow speed is a speed thatis slower than a fast speed, which is a speed that is faster than theslow speed; i.e., they are slower or faster relative to one another.)The first mode of operation may include moving at least one of the firstrobotic machine and the second robotic machine to determined locationsproximate to the target object and to each other; and the second mode ofoperation may include actuating one or more tools of at least one of thefirst robotic machine and the second robotic machine to accomplish thetask.

In one embodiment, a first robotic machine has a first set ofcapabilities for interacting with a surrounding environment, where thefirst robotic machine may receive a first sequence of sub-tasks relatedto the first set of capabilities of the first robotic machine; and asecond robotic machine has a second set of capabilities for interactingwith the surrounding environment. The second robotic machine may receivea second sequence of sub-tasks related to the second set of capabilitiesof the second robotic machine. The first and second robotic machines mayperform the first and second sequences of sub-tasks, respectively, toaccomplish a task that involves at least one of manipulating orinspecting a target object that is separate from the first and secondrobotic machines. The first and second robotic machines may coordinateperformance of the first sequence of sub-tasks by the first roboticmachine with performance of the second sequence of sub-tasks by thesecond robotic machine.

At least some of the sub-tasks may be sequential such that the secondrobotic machine may begin performance of a corresponding sub-task in thesecond sequence responsive to receiving a notification from the firstrobotic machine that the first robotic machine has completed a specificsub-task in the first sequence.

During operation, the system may include the first robotic machinehaving a first set of capabilities for interacting with a surroundingenvironment, the first robotic machine configured to receive a firstsequence of sub-tasks related to the first set of capabilities of thefirst robotic machine, and a second robotic machine having a second setof capabilities for interacting with the surrounding environment, thesecond robotic machine configured to receive a second sequence ofsub-tasks related to the second set of capabilities of the secondrobotic machine. The system may perform the first and second sequencesof sub-tasks to accomplish a task comprising at least one ofmanipulating or inspecting a target object, and may coordinateperformance of the first sequence of sub-tasks by the first roboticmachine with performance of the second sequence of sub-tasks by thesecond robotic machine.

FIG. 1 schematically illustrates an equipment system 50, a first roboticmachine 101, and a second robotic machine 102 according to oneembodiment. The equipment system may include first equipment 52 andsecond equipment 54 that may be mechanically interconnected to traveltogether along a route. The second equipment may be disposed in front ofthe first equipment in a direction of movement of the equipment system.In an alternative embodiment, the equipment system may include more thantwo interconnected pieces or items of equipment or only one item ofequipment. The first and second equipment are rail vehicles in theillustrated embodiment. In other embodiments, suitable mobile equipmentmay include automobiles, off-road equipment, or the like. In otherembodiments, suitable stationary equipment may include railroad tracks,roads, bridges, buildings, stacks, stationary machines, and the like.The first and second robotic machines may perform an assigned task onthe equipment. The first and second robotic machines collaborate (e.g.,work together) to accomplish the assigned task. The first and secondrobotic machines perform various sub-tasks semi-autonomously,autonomously (without direct control and/or supervision of a humanoperator), or manually under remote control of an operator. For example,the robotic machines may act based on instructions received prior tobeginning the sub-tasks. The assigned task may be completed upon thecompletion of the sub-tasks by the robotic machines. Whether the task ismanual, semi-auto, or autonomous is based in part on the sub-task, thecapabilities of the robotic machines, the target object, and the like.Further, inspection of a target object may determine whether thesub-task should be performed in a manual, semi-auto, or autonomousmanner.

In the illustrated embodiment, the first equipment has an air brakesystem 100 disposed onboard. The air brake system engages correspondingwheels 103 and may operate on a pressure differential within one or moreconduits 104 of the air brake system. When the pressure of a fluid, suchas air, in the conduits is above a designated threshold or when thepressure increases by at least a designated amount, air brakes 106engage corresponding wheels of the first equipment. (In certainequipment, such as certain rail vehicles, the air brakes may beconfigured to engage when the pressure of the fluid (e.g., air) in theconduits drops below a designated threshold.) Although only one airbrake is shown in FIG. 1, the air brake system may include several airbrakes. The conduit connects with a valve 108 that closes to retain thefluid (and fluid pressure) within the conduit. The valve can be openedto release (e.g., bleed) the fluid out of the conduit and the air brakesystem. (As noted, in certain equipment, once the pressure of the fluidin the conduit and air brake system drops by or below a designatedamount, the air brake engages the wheels.) The first equipment mayfreely roll while the air brake is disengaged, but is held fast whilethe air brake is engaged.

The valve can be actuated by manipulating (e.g., moving) a brake lever110. The brake lever can be pulled or pushed in a direction 111 to openand close the valve. The brake lever is an example of a target object.In an embodiment, releasing the brake lever may cause the valve toclose. For example, the brake lever may move under the force of a springor other biasing device to return to a starting position and force thevalve closed. In another embodiment, the brake lever may require anoperator or an automated system to return the brake lever to thestarting position to close the valve after bleeding the air brakesystem.

The second equipment may include its own air brake system 112 that maybe identical, or at least substantially similar, to the air brake systemof the first equipment. The second equipment may include a first hose114 (referred to herein as an air hose) that may fluidly connect to theconduit of the air brake system. The first equipment may include asecond hose 118 that may be fluidly connected to the same conduit. Thesecond hose extends from a front 128 of the first equipment, and thefirst hose extends from a rear 130 of the second equipment. The hosesmay connect to each other at a separable interface 119 to provide afluid path between the air brake system of the first equipment and theair brake system of the second equipment. Fluid may be allowed to flowbetween the air brake systems when the hoses may be connected. Fluidcannot flow between the air brake systems when the hoses aredisconnected. The first equipment has another second air hose at therear end 130 thereof, and the second equipment has another first airhose at the front end thereof.

The first equipment may include a hand brake system 120 disposed onboardthe first equipment. The hand brake system may include a brake wheel 122that may be rotated manually by an operator or an automated machine. Thebrake wheel is mechanically linked to friction-based hand brakes 124(e.g., shoes or pads) on the first equipment. Rotation of the brakewheel in a first direction causes the hand brakes to move towards andengage the wheels, setting the hand brakes. Rotation of the brake wheelin an opposite, second direction causes the hand brakes to move awayfrom and disengage the wheels, releasing the hand brakes. In analternative embodiment, the hand brake system may include a lever oranother actuatable device instead of the brake wheel. In the illustratedembodiment, the second equipment may include a hand brake system 121that may be identical, or at least substantially similar, to the handbrake system of the first equipment.

The first and second equipment may include mechanical couplers at boththe front ends and the rear ends of the equipment. The mechanicalcoupler at the rear end of the second equipment mechanically engages andconnect to the mechanical coupler at the front end of the firstequipment to interconnect or couple the equipment to each other. Thefirst equipment may be uncoupled from the second equipment bydisconnecting the mechanical couplers 126 that extend between the firstand second equipment.

The robotic machines may be discrete from the equipment system such thatneither robotic machine is integrally connected to the equipment system.The robotic machines may move relative to the equipment system tointeract with at least one of the first and/or second equipment. Each ofthe robotic machines has a specific set of affordances or capabilitiesfor interacting with the surrounding environment. Some examples ofcapabilities include flying, driving (or otherwise traversing along theground), lifting other objects, imaging (e.g., generating images and/orvideos of the surrounding environment), grasping an object, rotating,tilting, extending (or telescoping), retracting, pushing, pulling, orthe like. The first robotic machine has a first set of capabilities, andthe second robotic machine has a second set of capabilities.

In the illustrated embodiment, the first robotic machine may bedifferent than the second robotic machine, and has at least somedifferent capabilities than the first robotic machine. Thus, the secondset of capabilities of the second robotic machine may include at leastone capability that differs from the first set of capabilities of thefirst robotic machine or vice-versa. For example, the first roboticmachine in the illustrated embodiment has the capability to drive on theground via the use of multiple wheels 146. The first robotic machinealso has the capabilities to grasp and manipulate a target object 132 ona designated the equipment, such as the first equipment, using a roboticarm 210. The robotic arm may have the capabilities to rotate, tilt,lift, extend, retract, push, and/or pull the target object 132. Thefirst robotic machine may be referred to herein as a grasping roboticmachine. In the illustrated embodiment, the target object 132 may beidentified as the brake lever, but the target object 132 may be adifferent device on the first equipment depending on the assigned taskthat may be performed by the robotic machines.

The second robotic machine in the illustrated embodiment is an aerialrobotic machine (e.g., a drone) that has the capability to fly in theair above and/or along a side of the equipment system via the use of oneor more propellers 148. Although not shown, the robotic machine mayinclude wings that provide lift. The second robotic machine in FIG. 1may be referred to as an aerial robotic machine. The aerial roboticmachine may include an imaging device 150 that may be configured togenerate imaging data. Imaging data may include still images and/orvideo of the surrounding environment in the visual frequency range, theinfrared frequency range, or the like. Suitable imaging devices mayinclude an infrared camera, a stereoscopic 2D or 3D camera, a digitalvideo camera, or the like. Using the imaging device, the aerial roboticmachine has the capability to visually inspect designated equipment,including a target object thereof, such as to determine a position orstatus of the target object. The aerial robotic machine may not have thecapability to drive on the ground or grasp and manipulate a targetobject like a grounded grasping robotic machine. The grounded roboticmachine, on the other hand, may not have the capability to fly.

The robotic machines may perform an assigned task on one or both of thefirst and/or second equipment. For example, the robotic machines mayperform the assigned task on the first equipment, and then maysubsequently perform an assigned task on the second equipment. Theequipment system may include more than just the two items of equipmentshown in FIG. 1. The robotic machines may move along the equipmentsystem from one location or region of interest to another, and maydesignate new equipment and new target objects on which to performassigned tasks. Alternatively, the robotic machines may perform theassigned task on the first equipment and not on the second equipment, orvice-versa. The grasping and aerial robotic machines work together andcollaborate to complete the assigned task. The assigned task involves atleast one of the robotic machines engaging and manipulating the targetobject 132 on the designated equipment. In the illustrated embodiment,the grasping robotic machine may engage and manipulate the brake leverwhich defines the target object. The aerial robotic machine may flyabove the equipment system and inspect the target object using theimaging device. The aerial robotic machine also may use the imagingdevice to detect the presence of obstructions between the graspingrobotic machine and the target object. As discussed further herein, inone embodiment the imaging device may be used to help locate andnavigate the aerial robotic machine.

The aerial robotic machine and the grasping robotic machine shown inFIG. 1 are intended as examples. The first and second robotic machinesmay have other shapes and/or capabilities or affordances in otherembodiments, as shown and described herein. For example, the roboticmachines in one or more other embodiments may both be land-based and/ormay both have robotic arms 210 for grasping.

One assigned task may be for the robotic machines to bleed the air brakesystems of the respective equipment in the equipment system. Prior tothe equipment system starting to move from a stationary position, theair brake systems of each of the first and second equipment must bemanipulated to release the air brake. The brake lever is identified asthe target object. The grasping and aerial robotic machines collaborateto perform the assigned task. For example, the aerial robotic machinemay fly above the first equipment, locating and identifying the brakelever, and determining that the brake lever is in a non-actuatedposition requiring manipulation to release the air brake. The aerialrobotic machine informs the grasping robotic machine of the locationand/or status (e.g., non-actuated) of the target object and, optionally,the distance and orientation of the second robotic machine, or the armof the second robotic machine relative to the target object. Since thegrasping robotic machine traverses on the ground, the robotic machinemay be susceptible to obstructions blocking its path. The aerial roboticmachine optionally may inspect the path ahead of the ground roboticmachine and notify the ground robotic machine of any detected obstaclesbetween it and the target object (e.g., brake lever). The graspingrobotic machine receives and processes the information transmitted fromthe aerial robotic machine. The grasping robotic machine moves towardthe brake lever, engages the brake lever, and manipulates the brakelever by pulling or pushing the brake lever. The grasping roboticmachine and/or the aerial robotic machine determine whether the brakelever has been moved fully to the actuated position. Upon confirmationthat the air brake is released, the grasping robotic machine releasesthe brake lever. It may then move to the next item of equipment (e.g.,the second equipment) in the equipment system to repeat the brakebleeding task. Optionally, the robotic machines may implement one ormore follow up actions responsive to determining that the air brakesystem has or has not been released, such as by communicating with oneor more human operators, attempting to release the air brake systemagain, or identifying the first equipment having the air brake systemthat may be not released as requiring inspection, maintenance, orrepair. As discussed herein, at least one of the robotic machines mayconfirm completion of the sub-task using an acoustic sensor.

The robotic machines may perform additional or different tasks otherthan brake bleeding. For example, the robotic machines may be assignedthe task of setting and/or releasing the hand brakes of one or both ofthe first equipment and second equipment. The hand brakes may be set asa back-up to the air brake. When the equipment system stops, humanoperators may decide to set the hand brakes on only some of theequipment, such as the hand brakes on every fourth item of equipmentalong the length of the equipment system. One assigned task may be torelease the hand brakes on the equipment to allow the equipment systemto move along the route. In an embodiment, the aerial robotic machinemay fly along the equipment system to detect which of the variousequipment has hand brakes that need to be released, if any. The aerialrobotic machine may inspect the hand brakes along the equipment and/orthe positions of the brake wheels to determine which equipment needs tohave the hand brakes released. For example, the aerial robotic machinemay determine that the hand brake of the second equipment needs to bereleased, but the hand brake of the first equipment is not set. Theaerial robotic machine notifies the grasping robotic machine to actuatethe brake wheel of the second equipment, but not the brake wheel of thefirst equipment. The aerial robotic machine may provide otherinformation to the grasping robotic machine, such as distance from thegrasping robotic machine to the target object, the type and location ofobstacles detected in the path of the grasping robotic machine, and theconfiguration of the target object itself (not all items of equipmentmay be identical).

Upon receiving the communication from the aerial robotic machine, thegrasping robotic machine may move past the first equipment to the frontend of the second equipment. The grasping robotic machine manipulatesthe brake wheel, which represents the target object, by extending therobotic arm to the brake wheel, grasping the brake wheel, and thenrotating the brake wheel in a designated direction to release the handbrakes. After one or both robotic machines confirm that the hand brakesof the second equipment are released, the assigned task is designated ascomplete. The robotic machines may move to other the equipment (notshown) in the equipment system to perform an assigned task on otherequipment.

In another embodiment, the robotic machines may be assigned the task ofcoupling or uncoupling the first equipment relative to the secondequipment. The robotic machines may both be land-based (instead of theaerial machine shown in FIG. 1) and may perform the task by engaging andmanipulating the mechanical couplers of the equipment which representtarget objects. For example, the first robotic machine may engage thecoupler at the front of the first equipment, and the second roboticmachine may engage the coupler at the rear of the second equipment. Therobotic machines collaborate during the performance of the assigned taskin order to couple or uncouple the first equipment relative to eachother.

Yet another potential assigned task that may be assigned to the roboticmachines may be hose lacing. Hose lacing involves connecting (ordisconnecting) air hoses of the first equipment to each other to fluidlyconnect the air brake systems to itself. This closes an otherwise openfluidic circuit. For example, both robotic machines may have roboticarms like the robotic arm of the grasping robotic machine shown inFIG. 1. The first robotic machine may grasp an end of the second hose atthe front end of the first equipment, and the second robotic machinegrasps an end of the first hose at the rear end of the second equipment.The robotic machines communicate and collaborate to index and align(e.g., tilt, rotate, translate, or the like) the hoses of the two thefirst equipment with each other and then move the hoses relative to eachother to connect the hoses at the separable interface.

Although potential tasks for the robotic machines to perform on theequipment may be described with reference to FIG. 1, the roboticmachines may perform various other tasks on other equipment systems thatinvolve manipulating and/or inspecting a target object.

FIG. 2 illustrates an embodiment of the first robotic machine shown inFIG. 1. The grasping robotic machine is shown in a partially explodedview with several components (e.g., 202, 204, 208, and 222) displayedspaced apart from the robotic arm. The robotic arm may be mounted on amobile base 212 that includes wheels. The mobile base moves the roboticarm towards the target object of the equipment and transports the armfrom place to place. The robotic arm may move in multiple differentdirections and planes relative to the base under the control of a taskmanager via a controller 208. The controller drives the robotic arm tomove toward the corresponding target object (e.g., the brake lever shownin FIG. 1) to engage the target object and manipulate the target objectto perform the assigned task. For example, the controller may conveycommands in the form of electrical signals to actuators, motors, and/orother devices of the robotic arm that provide a kinematic response tothe received commands.

The controller represents hardware circuitry that may include,represent, and/or may be connected with one or more processors (e.g.,microprocessors, field programmable gate arrays, integrated circuits, orother electronic logic-based devices). The controller may include and/orbe communicatively connected with one or more digital memories, such ascomputer hard drives, computer servers, removable hard drives, etc. Thecontroller may be communicatively coupled with the robotic arm and themobile base by one or more wired and/or wireless connections that allowthe controller to dictate how and where the grasping robotic machinemoves. Although shown as a separate device that may be not attached tothe robotic arm or the mobile base, the controller may be mounted on therobotic arm and/or the mobile base.

The robotic arm may include an end effector 214 at a distal end 216 ofthe robotic arm relative to the mobile base. The end effector maydirectly engage the target object on the equipment to manipulate thetarget object. For example, the end effector may grasp the brake lever(shown in FIG. 1) to hold the lever such that subsequent movement of therobotic arm moves the brake lever with the arm. In the illustratedembodiment, the end effector has a claw 218 that may be controllable toadjust a width of the claw to engage and at least partially enclose thetarget object. The claw has two fingers 220 that may be movable relativeto each other. For example, at least one of the fingers may be movablerelative to the other finger to adjust the width of the claw and allowthe claw to grasp the target object. The end effector may have othershapes in other embodiments.

The grasping robotic machine may include a communication circuit 222.The communication circuit operably connects to the controller. Suitablecircuits may include hardware and/or software that may be used tocommunicate with other devices and/or systems, such as another roboticmachine (e.g., the second robotic machine shown in FIG. 1) configured tocollaborate with the robotic machine to perform the assigned task,remote servers, computers, satellites, and the like. The communicationcircuit may include a transceiver and associated circuitry (e.g., anantenna 224) for wireless bi-directional communication of various typesof messages, such as task command messages, notification messages, replymessages, feedback messages, or the like. The communication circuit maytransmit messages to specific designated receivers and/or broadcastmessages indiscriminately. In an embodiment, the communication circuitmay receive and convey messages to the controller prior to and/or duringthe performance of an assigned task. As described in more detail herein,the information received by the communication circuit from remotesources, such as another robotic machine collaborating with the roboticmachine, may be used by the controller to control the timing andmovement of the robotic arm during the performance of the assigned task.Although the communication circuit is illustrated as a box-shaped devicethat may be separate from the robotic arm and the mobile base, thecommunication circuit may be mounted on the robotic arm and/or themobile base.

The grasping robotic machine may include one or more sensors 202, 204,206 that monitor operational parameters of the grasping robotic machineand/or the target object that the robotic machine manipulates. Theoperational parameters may be communicated from the respective sensorsto the controller. The controller examines the parameters to makedeterminations regarding the control of the robotic arm, the mobilebase, and the communication circuit. In the illustrated example, therobotic machine may include an encoder sensor that converts rotaryand/or linear positions of the robotic arm into one or more electronicsignals. The encoder sensor can include one or more transducers thatgenerate the electronic signals as the arm moves. The electronic signalscan represent displacement and/or movement of the arm, such as aposition, velocity, and/or acceleration of the arm at a given time. Theposition of the arm may refer to a displaced position of the armrelative to a reference or starting position of the arm, and thedisplacement may indicate how far the arm has moved from the startingposition. Although shown separated from the robotic arm and mobile basein FIG. 2, the encoder sensor may be mounted on the robotic arm and/orthe mobile base in an embodiment.

The grasping robotic machine may also include an imaging sensor 206 thatmay be installed on the robotic arm. In an embodiment, the imagingsensor may be mounted on or at least proximate to the end effector. Forexample, the imaging sensor may include a field of view that encompassesat least a portion of the end effector. The imaging sensor moves withthe robotic arm as the robotic arm moves toward the brake lever. Theimaging sensor acquires perception information of a working environmentof the robotic arm. The perception information may include images and/orvideo of the target object in the working environment. The perceptioninformation may be conveyed to the controller as electronic signals. Thecontroller may use the perception information to identify and locate thetarget object relative to the robotic arm during the performance of theassigned task. Optionally, the perception information may bethree-dimensional data used for mapping and/or modeling the workingenvironment. For example, the imaging sensor may include an infrared(IR) emitter that generates and emits a pattern of IR light into theenvironment, and a depth camera that analyzes the pattern of IR light tointerpret perceived distortions in the pattern. The imaging sensor mayalso include one or more color cameras that operate in the visualwavelengths. The imaging sensor may acquire the perception informationat an acquisition rate of at least 15 Hz, such as approximately 30 Hz.Optionally, the imaging sensor may be a Kinect™ sensor manufactured byMicrosoft.

Suitable imaging sensors may include video camera units for capturingand communicating video data. Suitable images may be in the form ofstill shots, analog video signals, or digital video signals. Thesignals, particularly the digital video signals, may be subject tocompression/decompression algorithms, such as MPEG or HEVC. A suitablecamera may capture and record in a determined band of wavelengths oflight or energy. For example, in one embodiment the camera may sensewavelengths in the visible spectrum and, in another, the camera maysense wavelengths in the infrared spectrum. Multiple sensors may becombined in a single camera and may be used selectively based on theapplication. Further, stereoscopic and 3D cameras are contemplated forat least some embodiments described herein. These cameras may assist indetermining distance, velocity, and vectors to predict (and therebyavoid) collision and damage. For example, the camera may be deployedonboard a robotic machine to capture video data, for storage for lateruse. The robotic machine may act as a powered camera supporting object,such that the camera may be mobile. That is, the camera unit and itssupporting object may be capable of moving independent or separate frommovement of an operator or another robotic machine. The supportingobject may be a robotic machine, or an implement of the robotic machine.Suitable implements may include an extendable mast.

The camera unit may be connected or otherwise disposed onboard an aerialrobotic machine (e.g., a drone, helicopter, or airplane) to allow thecamera unit to fly, or the camera unit may be connected with orotherwise disposed onboard another ground-based or aquatic roboticmachine to allow the robot and camera relative movement. In oneembodiment, the camera supporting object is the first robotic machinecapable of at least one of remote control or autonomous movementrelative to the second robotic machine. The first robotic machine maytravel along a route ahead of the second robotic machine and maytransmit the image data back to the second robotic machine. This mayprovide an operator of the second robotic machine a view of the routewell in advance of the arrival of the second robotic machine. For veryhigh speed second robotic machines, the stopping distance may be beyondthe visibility provided from the vantage of the second robotic machine.The view from the first vehicle, then, may extend or supplement thatvisible range. In addition, the camera itself may be repositionable andmay have the ability to pan left, right, up and down, as well as theability to zoom in and out.

The camera unit or the supporting robotic machine can include a locatordevice that generates data used to determine its location. The locatordevice can represent one or more hardware circuits or circuitry thatinclude and/or are connected with one or more processors (e.g.,controllers, microprocessors, or other electronic logic-based devices).In one example, the locator device represents a global positioningsystem (GPS) receiver that determines a location of the camera unit, abeacon or other communication device that broadcasts or transmits asignal that is received by another component (e.g., the transportationsystem receiver) to determine how far the camera unit is from thecomponent that receives the signal (e.g., the receiver), a radiofrequency identification (RFID) tag or reader that emits and/or receiveselectromagnetic radiation to determine how far the camera unit is fromanother RFID reader or tag (e.g., the receiver), or the like. Thereceiver can receive signals from the locator device to determine thelocation of the locator device relative to the receiver and/or anotherlocation (e.g., relative to a vehicle or vehicle system). Additionallyor alternatively, the locator device can receive signals from thereceiver (e.g., which may include a transceiver capable of transmittingand/or broadcasting signals) to determine the location of the locatordevice relative to the receiver and/or another location (e.g., relativeto a vehicle or vehicle system).

The robotic machine may include a force sensor 204 that monitors forcesapplied by the robotic arm on the target object during the performanceof the assigned task as the robotic arm manipulates the target object.As used herein, the term “force” encompasses torque, such that theforces applied by the robotic arm on the target object described hereinmay or may not result in the target object twisting or rotating. Theforce sensor may communicate electronic signals to the controller thatrepresent the forces exerted by the robotic arm on the target object, asmonitored by the force sensor. The forces may represent forces appliedby the claw of the end effector on the target object. The sensed forcesmay represent those forces applied on various joints of the robotic armfor moving and maneuvering the arm.

Optionally, the robotic machine may include one or more other sensors inaddition to, or instead of one or more of, the sensors shown in FIG. 2.For example, the robotic machine may include an acoustic sensor thatdetects sounds generated during actuation of the brake lever (shown inFIG. 1) to determine whether the brake lever has been actuated torelease the air brake system. The acoustic sensor may detect contactbetween implements of one robotic machine and one or more of anotherrobotic machine, an implement of another robotic machine, anotherimplement of the first robotic machine, a portion of the target object,or another object that is neither a robotic machine (or its implements)or the target object. Feedback from the acoustic sensor may be used tocalibrate an implement's location and/or speed and/or force. Forexample, the sensing function may correlate a contact sound to indicatewhen a grip has been moved far enough to contact the target object. Atthat moment, the task manager may index the robotic machine's implement(or the robotic machine's location) relative to the target object oranother object. The task manager may base task instruction on such indexinformation. If the sub-task commanded an implement to contact anobject, that sub-task may be considered to be fulfilled so that thesequentially next sub-task may start. The magnitude of the acousticsignal may correlate to the force of impact of the implement with thetarget object. Adjustments to movement speed and movement force may bemade based at least in part on the signal magnitude. An operating modemay be initiated if the task manager is unsure of location of animplement or robotic machine to intentionally make contact to generatean acoustic signal, and therefore ascertain relative locations or verifylocations.

The robotic machine may include one or more other sensors that mayfunction similarly to the uses set forth for the acoustic sensor, asmodified by the application and the sensor type. Suitable sensors maymonitor the speed of the mobile robotic machine and/or the speed atwhich an implement, such as a robotic arm, moves relative to the roboticmachine base. In one embodiment, at least one robotic machine may definea plurality of zones of movement for an implement. Such zones may bedesignated in such a way that the task manager, or the robotic machine,may behave differently based on triggers or activities associated withone of the zones that differs from behavior in other zones. In oneexample, a zone may be a potential contact zone insofar as an implementof a robotic machine may be operating in such potential contact zone andthat if there is an object, such as a target object or an obstacle, insuch potential contact zone the implement may impact or act on thatobject. The task manager may be apprised of, via one or more sensors,the presence or absence of such an object in such a zone.

In one embodiment, differences in acoustic signals are modeled andassociated with various types of activities. A human voice that issensed may indicate the presence of a human within a potential contactzone. As such, the task manager may preclude some sub-tasks, such asmovement of the robotic machine or its implement, until verification canbe made that no human is located in the potential contact zone.Verification might be made by a manual indication that all persons havevacated such potential contact zone. Or, a second set of sensors mayconfirm that, for example, a signaling tag worn by a person is notpresent in such potential contact zone prior to allowing movement of animplement in such potential contact zone. Alternatively, a lock outsystem may be employed such that if a lock out tag, or equivalent, isset for a particular zone the robotic machine may not move or may notmove an implement into or through such potential contact zone until acorresponding lock out tag is removed.

Movement in other zones may be allowed even while other zones have oneor more tasks and activities constrained. In one embodiment, if arobotic machine has four lateral zones defined as forward, rearward,left and right and an obstacle or equivalent (such as a lock out tag, ora detected human (via voice or image)) is sensed in the forward zone,the robotic machine may move itself and/or an implement into one of thethree remaining zones while avoiding movement or activity in the forwardzone.

In one embodiment, the task includes an inspection plan including thevirtual 3D travel path of and/or about an asset. In a first operatingmode, one or more robotic machine may travel to a location proximate tothe target object in the real world based on, for example, globalpositioning system (GPS) coordinates of the asset in comparison to GPScoordinates of the robot and, on arrival, align or index to a virtuallycreated 3D model. In a second travel mode, the robotic machine maytravel and align with the target object based on sensor input (otherthan GPS) relative to the 3D model. That is, once a robotic machine hasarrived at a desired start location the robotic machine may move alongthe real travel path from the start location to an end location in anautonomous or semi-autonomous fashion based at least in part on the 3Dmodel and environmental sensors. This may be useful where the targetobject is very large, e.g., a section of road, a railway, a bridge, or abuilding. Specific areas of interest about the target object may bemonitored and evaluated dynamically by the robotic machine.

During travel, the robotic machine may stop, pause, slow down, speed-up,maintain speed, etc., capture images, as well as sense for other data(e.g., temperature, humidity, pressure, etc.) at various regions ofinterest (ROI) designated by the task manager. For each ROI, the virtual3D model may include three-dimensional coordinates (e.g., X, Y, and Zaxis coordinates) at which the robotic machine is to be located forperforming a particular sub-task. In addition to a location in threedimensional space, each ROI may include a perspective with respect to asurface of the target object at which the sub-task may dictate thecapture of data or images, field of view of a camera, orientation, andthe like. To execute sub-tasks, the robotic machine may use multiplemodels. Suitable models may include a model of the world for safeautonomous navigation to and from a work site, and another model of thetarget object which contains the location of regions of interest. Basedon the model of the world the task manager may determine how to orientthe first robotic machine or the second robotic machine relative to eachother, the target object, and/or the surrounding environment. Based onthe model of the asset, each robotic machine can execute sub-tasks atregions of interest. The first and second robotic machines may move as aconsist.

FIG. 3 is a schematic block diagram of a control system 230 forcontrolling first and second robotic machines 301, 302 to collaborate inperforming an assigned task on the equipment. The control system mayinclude the first and second robotic machines and a task manager 232.The task manager may be located remote from the first robotic machineand/or the second robotic machine. The task manager may communicate withthe robotic machines to provide instructions to the robotic machinesregarding the performance of an assigned task that involves manipulatingand/or inspecting a target object on the equipment. The first and secondrobotic machines may use the information received from the task managerto plan and execute the assigned task.

The first and second robotic machines may or may not be the graspingrobotic machine and the aerial robotic machine, respectively, of theembodiment shown in FIG. 1. For simplicity of description, FIG. 3 doesnot illustrate all of the components of the robotic machines, such asthe robotic arm and the propellers (both shown in FIG. 1). Each of therobotic machines may include a communication circuit and a controller asdescribed with reference to FIG. 2. Each of the controllers may includeone or more processors 248. The controllers may be optionallyoperatively connected to respective digital memory devices 252.

The task manager may include a communication circuit 234, at least oneprocessor 238, and a digital database 236, which may represent or becontained in a digital memory device (not shown). The processor may beoperatively coupled to the database and the communication circuit. Thetask manager may be or include a computer, a server, an electronicstorage device, or the like. The database may be, or may be containedin, a tangible and non-transitory (e.g., not a transient signal)computer readable storage medium. The database stores informationcorresponding to multiple robotic machines in a group of roboticmachines that may include the first and second robotic machines. Forexample, the database may include a list identifying the roboticmachines in the group and providing capabilities or affordancesassociated with each of the robotic machines in the list. The databasemay also include information related to one or more potential assignedtasks, such as a sequence of sub-tasks to be performed in order toaccomplish or complete the assigned task. Optionally, the database mayinclude information about one or more items of equipment on which anassigned task is to be performed, such as information about types andlocations of various potential target objects on the equipment to bemanipulated in the performance of an assigned task. The processor may beaccess the database to retrieve information specific to an assignedtask, the equipment on which the assigned task is to be performed,and/or a robotic machine that may be assigned to perform the task.Although shown as a single, unitary hardware device, the task managermay include multiple difference hardware devices communicativelyconnected to one another. For example, in an embodiment, the taskmanager may be one or more servers located at a data center, a railroaddispatch location, a control center, or the like.

The task manager may communicate with the first and second roboticmachines via the transmission of messages from the communication circuitto the communication circuits of the robotic machines. For example, thetask manager may communicate messages wirelessly in the form ofelectromagnetic radio frequency signals. The first and second roboticmachines may transmit messages to the task manager via the respectivecommunication circuits. The robotic machines may be also able tocommunicate with each other using the communication circuits. Forexample, the robotic machines may transmit status-containingnotification messages back and forth as the robotic machines collaborateto perform an assigned task in order to coordinate the actions of therobotic machines to perform the assigned task correctly and efficiently.Time sensitive networks may be used to coordinate activities requiring ahigh degree of precision in the coordination.

FIG. 4 illustrates a flow diagram 400 showing interactions of the taskmanager and the first and second robotic machines of FIG. 3 to controland coordinate the performance of an assigned task by the roboticmachines on the equipment according to an embodiment. The flow diagramis divided into a first column 402 listing actions or steps taken by thetask manager, a second column 404 listing actions or steps taken by thefirst robotic machine, and a third column 406 listing actions or stepstaken by the second robotic machine. At step 408, the task managergenerates a task that step involves manipulating and/or inspecting atarget object on the equipment. Various example tasks may be describedabove with reference to FIG. 1, including brake bleeding of an air brakesystem, setting or releasing a hand brake, inspecting a position of abrake actuator (e.g., a lever, a wheel, or the like), mechanicallycoupling or uncoupling the equipment relative to another the equipment,hose lacing to connect or disconnect an air hose of the equipment to anair hose of another the equipment, or the like. Optionally, the task maybe a scheduled task, and the task manager generates a sub-taskresponsive to the task being due to be performed. Alternatively, thetask manager may generate a sub-task upon receiving a request that stepthe task be performed, such as from a user interface connected to thetask manager or from a remote source via a communicated message.

At step 410, the task manager determines which robotic machines (e.g.,robots, drones) to employ to work together to perform the designatedtask. For example, the database of the task manager shown in FIG. 3 maystore information about the designated task, including which sub-goalsor sub-tasks may be required or at step least helpful to accomplish thedesignated task efficiently. The sub-tasks may be steps in the processof performing the task, such as moving toward a target object, engaginga certain portion of the target object, and applying a specific force onthe target object to move the target object for a specified distance ina specified direction. The database may also store information about agroup of multiple robotic machines including, but not limited to, thefirst and second robotic machines shown in FIG. 3. The information aboutthe robotic machines may include capability descriptions associated witheach robotic machine in the group. The capability descriptions mayinclude a list of the capabilities or affordances of the correspondingrobotic machine, such as the capability to grasp and pull a lever. Aprocessor of the task manager may perform an affordance analysis bycomparing the sub-tasks associated with the designated task to thecapability descriptions of the available robotic machines in the group.The processor determines a level of suitability of each of the availablerobotic machines to the specific sub-tasks for the designated task. Theavailable robotic machines may be ranked according to the level ofsuitability. For example, robotic machines that are capable of flyingwould rank highly for sub-tasks involving flight, but robotic machinesincapable of flight would rank low for the same sub-tasks. The processormay rank the robotic machines, and determine the robotic machines toemploy for the designated task based on the highest ranking availablerobotic machines for the sub-tasks.

In an example, the designated task involves manipulating a brakeactuator, which generally requires a robotic arm engaging the brakeactuator to move the brake actuator. If none of the available roboticmachines that have robotic arms are tall enough or able to extend farenough to engage the brake actuator, the processor of the task managermay select one of the highest ranking available robotic machines thathas a robotic arm. The processor may analyze the rest of the availablerobotic machines to determine which robotic machines are able to assistthe robotic machine with the robotic arm. The processor may select arobotic machine that is capable of lifting the robotic machine havingthe robotic arm, such that the robotic arm is able to engage andmanipulate the brake actuator when lifted. Thus, the task manager mayselect the robotic machines to employ for performing the designated taskbased on the suitability of the robotic machines to perform requiredsub-tasks as well as the suitability of the robotic machines tocoordinate with each other.

At step 412, the task manager assigns a first sequence of sub-tasks to afirst robotic machine and assigns a second sequence of sub-tasks to asecond robotic machine. Although not shown in the illustratedembodiment, the task manager may assign sub-tasks to more than tworobotic machines in other embodiments. For example, some tasks mayrequire three or more robotic machines working together to complete. Thesequences of sub-tasks may be specific steps or actions to be performedby the corresponding robotic machines in a specific order. The sub-tasksmay be similar to instructions. The performance of all of the sub-tasksby the corresponding robotic machines in the correct order may completeor accomplish the assigned task. The first and second sequences ofsub-tasks may be coordinated with each other. The first sequence ofsub-tasks (to be performed by the first robotic machine) in anembodiment may be at least partially different than the second sequenceof sub-tasks (to be performed by the second robotic machine). Forexample, at least some of the sub-tasks in the first sequence may differfrom at least some of the sub-tasks in the second sequence, orvice-versa. Some sub-tasks may be common to both the first and secondsequences, such that the sub-tasks may be performed by both roboticmachines. In an embodiment, the first and second sequences of sub-tasksdelineate specific steps or actions to be performed by the respectiverobotic machines and provide timing information. For example, the firstsequence may specify an order that the sub-tasks are to be performedrelative to each other and relative to the sub-tasks in the secondsequence to be performed by the second robotic machine. Thus, the firstsequence may specify that after completing a given sub-task, the firstrobotic machine is to wait until receiving a notification from thesecond robotic machine that a specific sub-task in the second sequencehas been completed before starting a subsequent sub-task in the firstsequence.

The first and second sequences of sub-tasks may be generated by the atleast one processor of the task manager after determining which roboticmachines to use, or may be pre-stored in the database or another memorydevice. For example, the database may store a list of potential assignedtasks and sequences of sub-tasks associated with each of the assignedtasks. Thus, upon generating the task and/or determining the roboticmachines, the processor may access the database to select the relevantsequences of sub-tasks associated with the assigned task.

At step 414, the task manager may transmit the first sequence ofsub-tasks to the first robotic machine and the second sequence ofsub-tasks to the second robotic machine. For example, the first andsecond sequences may be transmitted in respective command messages viathe communication circuit of the task manager. The task managercommunicates a command message containing the first sequence ofsub-tasks to the first robotic machine and another command messagecontaining the second sequence to the second robotic machine.

At step 416, the first robotic machine receives the command messagecontaining the first sequence of sub-tasks. At step 418 the secondrobotic machine receives the command message containing the secondsequence of sub-tasks. The communication circuits of the first andsecond robotic machines shown in FIG. 3 receive the command messages andcommunicate the contents to the respective controllers. At step 420, thefirst robotic machine generates a sub-task performance plan based on thefirst sequence of sub-tasks. The sub-task performance plan may be motionplanning by the first robotic machine that yields various motions,actions, and forces, including torques, to be produced by differentcomponents of the first robotic machine to perform the sub-tasks in thefirst sequence. The processors of the first robotic machine may usedynamic movement primitives to generate the performance plan. In anembodiment in which the first robotic machine may include the roboticarm (shown in FIG. 2), the sub-task performance plan may include amotion trajectory that plans the movement of the robotic arm from astarting position to the target object on the equipment. The sub-taskperformance plan may also provide a prescribed approach orientation ofthe robotic arm, including the claw of the end effector (shown in FIG.2), as the robotic arm approaches and engages the target object, andplanned forces to be exerted by the robotic arm on the target object tomanipulate the target object (e.g., a planned pulling force, directionof the force, and/or distance along which the force may be applied). Inother embodiments, the sub-task performance plan may specify coordinatesand/or distances that the first robotic machine, or components thereof,moves. At step 422, the second robotic machine generates a sub-taskperformance plan based on the second sequence of sub-tasks. The sub-taskperformance plan of the second robotic machine may be different than thesub-task performance plan of the first robotic machine, but may begenerated in a similar manner to the sub-task performance plan of thefirst robotic machine.

At step 424, the first robotic machine commences execution of the firstsequence of sub-tasks. At step 426, the second robotic machine commencesexecution of the second sequence of sub-tasks. Although steps 424 and426 are shown side-by-side in the diagram 400 of FIG. 4, the first andsecond robotic machines may or may not perform the respective sub-tasksduring the same time period. Depending on the sequences of sub-tasks ascommunicated by the task manager, the first robotic machine may beordered to start performing the sub-tasks in the first sequence beforeor after the second robotic machine starts performing the secondsequence of sub-tasks.

In an embodiment, the first and second robotic machines may coordinateperformance of the respective sequences of sub-tasks to accomplish theassigned task. Thus, the performance of the first sequence of sub-tasksby the first robotic machine may be coordinated with the performance ofthe second sequence of sub-tasks by the second robotic machine. In anembodiment, the first and second robotic machines coordinate bycommunicating directly with each other during the performances of thesub-tasks. At step 428, the first robotic machine provides a statusnotification to the second robotic machine. The status notification maybe a message communicated wirelessly as electromagnetic RF signals fromthe communication circuit of the first robotic machine to thecommunication circuit of the second robotic machine. The second roboticmachine receives the status notification at step 434. The statusnotification may inform the second robotic machine that the firstrobotic machine has started or completed a specific sub-task in thefirst sequence. The second robotic machine processes the received statusnotification and may use the status notification to determine when tostart performing certain sub-tasks in the second sequence. For example,at least some of the sub-tasks in the first and second sequences may besequential, such that the second robotic machine may begin performanceof a corresponding sub-task in the second sequence responsive toreceiving the notification from the first robotic machine that the firstrobotic machine has completed a specific sub-task in the first sequence.Other sub-tasks in the first and second sequences may be performedconcurrently by the first and second robotic machines, such that thetime period that the first robotic machine performs a given sub-task inthe first sequence at least partially overlaps the time period that thesecond robotic machine performs a given sub-task in the second sequence.For example, both robotic machines may concurrently move towards theequipment. In another example, the first robotic machine may extend arobotic arm towards the target object of the equipment concurrently withthe second robotic machine lifting the first robotic machine.Coordinated and concurrent actions by the robotic machines may enhancethe efficiency of the performance of the assigned task on the equipment.

The first robotic machine may transmit a status notification uponstarting and/or completing each sub-task in the first sequence, or maytransmit status notifications only upon starting and/or completingcertain designated sub-tasks of the sub-tasks in the first sequence,which may be identified in the command message sent from the taskmanager. At step 430, the second robotic machine provides a statusnotification to the first robotic machine. The status notification fromthe second robotic machine may be similar in form and/or function to thestatus notification sent from the first robotic machine at step 428. Thefirst robotic machine receives the status notification from the secondrobotic machine at step 432.

At steps 436 and 438, respectively, the first and second roboticmachines complete the performances of the first and second sequences ofsub-tasks. At step 440, the first robotic machine transmits a taskcompletion notification to the task manager that the first sequence maybe completed. At step 442, the second robotic machine transmits a taskcompletion notification to the task manager that the second sequence wascompleted. The first and second robotic machines may also notify eachother upon completing the sequences of sub-tasks, and optionally mayonly transmit a single task completion notification to the task managerinstead of one notification from each robotic machine. The one or morenotifications inform the task manager that the assigned task wascompleted. At step 444, the task manager receives and processes the oneor more notifications. The notification may also provide feedbackinformation to the task manager, such as force parameters used tomanipulate the target object on the equipment and other parametersmonitored and recorded during the performance of the sub-tasks. Theinformation received in the task completion notification may be used bythe task manager to update the information provided in future commandmessages to robotic machines, such as the sequences of sub-taskscontained in the command messages. Upon receiving the task completionnotification, the task manager may generate a new task for the same ordifferent robotic machines. For example, the task manager may assign thesame task to the same robotic machines for the robotic machines toperform the task on another the equipment in the same or a different theequipment system. Thus, the first and second robotic machines may becontrolled to move along a length of the equipment system to perform theassigned task on multiple items of equipment of the equipment system.Alternatively, the task manager may control the same or differentrobotic machines to perform a different assigned task on the sameequipment after completion of a first assigned task on the equipment.

FIG. 5 is a block flow diagram 500 showing a first sequence 502 ofsub-tasks assigned to a first robotic machine and a second sequence 504of sub-tasks assigned to a second robotic machine for performance of anassigned task on the equipment according to an embodiment. The diagrammay be described with reference to the grasping robotic machine and theaerial second robotic machine shown in FIG. 1. In the illustratedembodiment, the assigned task may be to manipulate a brake actuator ofthe first equipment of FIG. 1, such as the brake wheel of the hand brakesystem or the brake lever of the air brake system. The first and secondsequences 502, 504 of sub-tasks may be transmitted to the grasping andaerial robotic machines by the task manager (shown in FIG. 3).

The first sub-task in the second sequence 504 at step 506 commands theaerial robotic machine to fly along the first equipment, such as aboveor along a side of the first equipment. At step 508, the aerial roboticmachine identifies the target object, which may be the brake actuator.The aerial robotic machine may use the imaging device to generate imagedata of the surrounding environment including the first equipment. Oneor more processors of the aerial robotic machine may provide imageanalysis to identify the brake actuator in the image data captured bythe imaging device. The aerial robotic machine at step 510 determines aposition of the target object, such as a location of the brake actuatorrelative to the first equipment and/or whether the brake actuator is inan actuated or non-actuated position relative to the first equipment.For example, if the aerial robotic machine determines that the brakeactuator is already in an actuated position, there may be no need tomanipulate the brake actuator. The actuated position may represent, forexample, a pulled position of the brake lever that indicates that stepthe air brake may be bled, or a rotated position of the brake wheel thatindicates that the hand brakes may be released. The aerial roboticmachine determines the position of the target object using imageanalysis. At step 512, the aerial robotic machine transmits a statusnotification to the grasping robotic machine. The status notificationmay be similar to the status notifications described at step 428 and 430in FIG. 4. The status notification provides the position of the targetobject to the grasping robotic machine.

At step 514, the grasping robotic machine receives and processes thestatus notification transmitted by the aerial robotic machine. Theprocessors (e.g., the processors shown in FIG. 3) of the robotic machinedetermine whether or not to approach the first equipment. For example,since the task may be to actuate a brake actuator of the firstequipment, if the status notification indicates that the brake actuatormay be already in the actuated position, then there may be no need tomanipulate the brake actuator. Thus, if the brake actuator is in theactuated position, the grasping robotic machine at step 516 does notapproach the equipment. Instead, the robotic machine may move towardsanother equipment on which the robotic machine may be assigned toperform a task. If, on the other hand, the brake actuator is determinedby the aerial robotic machine to be in a non-actuated position, then thegrasping robotic machine at step 518 approaches the equipment. Forexample, the robotic machine may drive or otherwise move along theground towards the equipment and proximate to the brake actuatorthereof.

At step 520, the grasping robotic machine identifies the target objecton the first equipment. The robotic machine may identify the targetobject using image analysis based on image data captured by the imagingsensor (shown in FIG. 2). The image analysis may determine the location,tilt, size, and other parameters of the target object. At step 522, therobotic machine extends towards the target object. For example, therobotic arm (shown in FIG. 1) may extend from a retracted position to anextended position by generating torques at step various joints along thearm and/or by telescoping. At step 524, the robotic machine grasps andengages the target object. For example, the claw of the end effector(shown in FIG. 2) may grasp the brake actuator that step defines thetarget object. At step 526, the robotic machine manipulates the targetobject. In an embodiment, the robotic arm manipulates the brake actuatorby moving the brake actuator from the non-actuated position to theactuated position. The robotic arm may rotate the brake wheel, translatethe brake lever, or the like, to move the brake actuator to the actuatedposition. Upon manipulating the brake actuator, the grasping roboticmachine at step 528 generates and transmits a status notification to theaerial robotic machine. The status notification informs the aerialrobotic machine that the target object has been manipulated.

At step 530, the aerial robotic machine receives and processes thestatus notification received from the grasping robotic machine.Responsive to being notified that the target object has beenmanipulated, the aerial robotic machine at step 532 verifies whether ornot the target object is fully actuated (e.g., has been fully andsuccessfully manipulated to complete the task). For example, for a taskto bleed air brakes, the verification may include validating that thevalve of the air brake system has been sufficiently opened such that asufficient amount of air has been released from the air brake system toallow the brake to move to a released state. Verification by the aerialrobotic machine may be accomplished by various methods, includingaudibly recording the release of air using an audible sensor, detectingmovement of the brakes to the released state using the imaging device,detecting that the brake lever is in a designated actuated positionusing the imaging device, and/or the like. Although not shown, thegrasping robotic machine may also verify whether the brake lever isfully actuated, such as by using the encoder to detect that the roboticarm has moved the lever to a designated location, using the force sensorto detect the force exerted on the brake lever, and/or the like.

After the verification step, the aerial robotic at step 534 transmits astatus notification to the grasping robotic machine, which may bereceived by the robotic machine at 538. The status notification containsthe results of the verification step, such as whether or not the brakeactuator has been fully actuated and the task has been successfullycompleted. If the status notification indicates that the brake actuatoris not in the actuated position, then the grasping robotic machine mayreturn to step 526 and manipulate the brake actuator a second time. If,on the other hand, the status notification indicates that the brakeactuator is actuated and the task has been successfully completed, thenthe grasping robotic machine may, at step 540, control the robotic armto release the brake actuator that defines the target object. At step542, the robotic arm retracts away from the target object, returning toa retracted position on the robotic machine. At step 544, the graspingrobotic machine moves on the ground away from the first equipment.

At step 536, the aerial robotic machine flies away from the firstequipment. For example, the aerial robotic machine may fly towards asubsequent item of equipment (e.g., the second equipment shown in FIG.1), and may repeat the first sequence at step 504 of sub-tasks for thesecond equipment. Although not shown, the aerial robotic machine mayprovide guidance for the grasping robotic machine as the graspingrobotic machine moves along the ground. The aerial robotic machineprovides guidance by monitoring for obstacles along a path of therobotic machine, and may notify the robotic machine if the aerialrobotic machine detects the presence of an obstacle.

As shown in FIG. 5, the two robotic machines collaborate during theperformance of the respective sub-tasks to accomplish the assigned task.The aerial robotic machine may inspect the target object, verifyactuation of the target object, and/or provide guidance for the graspingrobotic machine. The grasping robotic machine may engage and manipulatethe target object on the equipment. It may be recognized that at leastsome of the sub-tasks in the first and second sequences 502, 504 may besequential, and at least some may be concurrent. For example, thegrasping robotic machine does not approach the equipment at step 518until receiving the notification from the aerial robotic machine that istransmitted at step 512. The aerial robotic machine may perform thesub-task of flying away from the equipment at step 536 concurrently tothe grasping robotic machine releasing the target object, retractingfrom the target object, and/or moving away from the equipment at steps540-544.

FIG. 6 is a perspective view of two robotic machines collaborating toperform an assigned task on the first equipment according to anotherembodiment. The task involves pulling a brake lever of the firstequipment. A first robotic machine 601 may be a grasping robotic machinethat may include a robotic arm 604 and may be at least similar to thegrasping robotic machine shown in FIG. 2. In the illustrated embodiment,the grasping robotic machine may be too short and cannot extend farenough to properly reach and engage the brake lever. A second roboticmachine 602 may be a lifting robotic machine that may be collaboratewith the grasping robotic machine 601 to perform the assigned task. Thelifting second robotic machine may include a body 606 and a platform 608that may be movable vertically relative to the body. The body mayinclude continuous tracks 610 for allowing the second robotic machine tonavigate obstacles and rocky terrain. The platform may be coupled to thebody via a telescoping tower 612 that may be used to lift and lower theplatform relative to the body.

In the illustrated embodiment, the assigned task may be performed by thelifting second robotic machine and the grasping first robotic machineeach performing a respective sequence of sub-tasks (e.g., assigned by atask manager). For example, a first sequence of sub-tasks for thegrasping robotic machine may include driving onto the platform of thelifting robotic machine, when the platform may be in a lowered, startinglocation at or proximate to the ground. A second sequence of sub-tasksfor the lifting robotic machine may include lifting the grasping roboticmachine on the platform vertically upwards from the starting location toa lifted location that may be disposed more proximate to the brake lever(or another target object) than when the grasping robotic machine is inthe starting location. Once the grasping robotic machine is in thelifted location, the robotic arm extends to the brake lever, grasps thebrake lever, and manipulates the brake lever by pushing or pulling in adesignated direction. After manipulating the brake lever and verifyingthat the brake lever manipulation has been successfully completed, thegrasping robotic machine sends a notification to the lifting roboticmachine. Responsive to receiving the notification, the lifting roboticmachine lowers the platform, and the grasping robotic machine thereon,back to the starting location on or proximate to the ground.Alternatively, the lifting robotic machine may lower the platform to anintermediate location, and may carry the grasping robotic machine toanother the equipment for performance of the same or a similar task onthe other the equipment. An additional robotic machine, such as theaerial robotic machine shown in FIG. 1, optionally may be employed tocollaborate with the robotic machines 601, 602 in the performance of theassigned task.

FIG. 7 is a perspective view of two robotic machines 701, 702collaborating to perform an assigned task on a first equipment accordingto yet another embodiment. The assigned task involves connecting asecond air hose of the first equipment to a corresponding first air hoseof a second equipment adjacent to the first equipment. The task may bereferred to as hose lacing. The two robotic machines may both begrasping robotic machines at least similar to the grasping roboticmachine in FIG. 2. A first robotic machine 701 and a second roboticmachine 702 include respective first and second robotic arms 704, 706,each similar to the robotic arm shown in FIG. 2.

In an embodiment, the first robotic machine performs the first sequenceof sub-tasks by locating and identifying the second air hose of thefirst equipment, then extending the first robotic arm and grasping thesecond air hose. The second robotic machine performs the second sequenceof sub-tasks by locating and identifying the first air hose of thesecond equipment, then extending the second robotic arm and grasping thefirst air hose. The second sequence of sub-tasks may instruct the secondrobotic machine to adjust an orientation of an end 708 of the first airhose to a designated orientation relative to the first equipment. Thefirst sequence of sub-tasks may instruct the first robotic machine toadjust both the position and orientation of an end 710 of the second airhose. The first robotic arm of the first robotic machine may moverelative to the second robotic arm of the second robotic machine towardsthe first air hose in order to connect the end of the second air hose tothe end of the first air hose. One or both of the robotic arms may moveand/or rotate to secure the hoses to one another, such as via abayonet-style connection. The robotic machines may coordinate themovements by communicating directly with each other during theperformance of the assigned task. The robotic machines may also beconfigured to collaborate to disconnect the air hoses in anotherassigned task.

In an embodiment, a system (e.g., a control system) may include a firstrobotic machine, a second robotic machine, and a task manager. The firstrobotic machine has a first set of capabilities for interacting with asurrounding environment. The second robotic machine has a second set ofcapabilities for interacting with the surrounding environment. The taskmanager has one or more processors. The task manager may select thefirst and second robotic machines from a group of robotic machines toperform a task that involves at least one of manipulating or inspectinga target object of the equipment that may be separate from the first andsecond robotic machines. The task manager may select the first andsecond robotic machines to perform the task based on the first andsecond sets of capabilities of the respective first and second roboticmachines. The task manager assigns a first sequence of sub-tasks to thefirst robotic machine for performance by the first robotic machine and asecond sequence of sub-tasks to the second robotic machine forperformance by the second robotic machine. The first and second roboticmachines may coordinate performance of the first sequence of sub-tasksby the first robotic machine with performance of the second sequence ofsub-tasks by the second robotic machine to accomplish the task.

Optionally, the first and second sets of capabilities of the first andsecond robotic machines each include at least one of flying, driving,diving, lifting, imaging, grasping, rotating, tilting, extending,retracting, pushing, pulling, welding, cutting, polishing, spraying,and/or the like. The second set of capabilities of the second roboticmachine may include at least one capability that differs from the firstset of capabilities of the first robotic machine. The task may includeactuating a lever to open a valve of the equipment.

Optionally, the first and second robotic machines coordinate performanceof the first sequence of sub-tasks by the first robotic machine with theperformance of the second sequence of sub-tasks by the second roboticmachine by communicating directly with each other. Responsive tocompleting a corresponding sub-task in the first sequence, the firstrobotic machine may notify the second robotic machine that thecorresponding sub-task is complete. At least some of the sub-tasks maybe sequential such that the second robotic machine may begin performanceof a corresponding sub-task in the second sequence responsive toreceiving a notification from the first robotic machine that the firstrobotic machine has completed a specific sub-task in the first sequence.The first robotic machine may perform at least one of the sub-tasks inthe first sequence concurrently with performance of at least one of thesub-tasks in the second sequence by the second robotic machine.

Optionally, the task manager may access a database that storescapability descriptions corresponding to each of the robotic machines inthe group of robotic machines. The task manager may select the first andsecond robotic machines to perform the task instead of other roboticmachines in the group based on a suitability of the capabilitydescriptions of the first and second robotic machines to the task. Thefirst robotic machine may perform the first sequence of sub-tasks bylifting the second robotic machine from a starting location to a liftedlocation such that the second robotic machine in the lifted location maybe disposed more proximate to the target object of the equipment thanwhen the second robotic machine is in the starting location. Responsiveto receiving a notification from the second robotic machine that atleast one of manipulation or inspection of the target object iscomplete, the first robotic machine may lower the second robotic machineback to the starting location.

Optionally, the first robotic machine may perform the first sequence ofsub-tasks by flying above or along a side of the equipment, identifyingthe target object of the equipment, determining a position of the targetobject, and providing a notification to the second robotic machine ofthe position of the target object. The second robotic machine performsthe second sequence of sub-tasks by moving on the ground to theequipment proximate to the target object, extending a robotic arm of thesecond robotic machine to the target object, engaging and manipulatingthe target object, releasing the target object, and retracting therobotic arm.

Optionally, multiple items of equipment may be coupled together. Thefirst robotic machine may perform the first sequence of sub-tasks byextending a robotic arm of the first robotic machine and grasping atarget object of the first equipment. The second robotic machine mayperform the second sequence of sub-tasks by extending a robotic arm ofthe second robotic machine to a target object of the second equipmentadjacent to the first equipment. The robotic arms of the first andsecond robotic machines move relative to one another with thecorresponding target objects to at least one of connect or disconnectthem.

In an embodiment, a system (e.g., a control system) includes a firstrobotic machine and a second robotic machine. The first robotic machinehas a first set of capabilities for interacting with a surroundingenvironment. The first robotic machine may receive a first sequence ofsub-tasks related to the first set of capabilities of the first roboticmachine. The second robotic machine has a second set of capabilities forinteracting with the surrounding environment. The second robotic machinemay receive a second sequence of sub-tasks related to the second set ofcapabilities of the second robotic machine. The first and second roboticmachines may perform the first and second sequences of sub-tasks,respectively, to accomplish a task that involves at least one ofmanipulating or inspecting a target object of the equipment that may beseparate from the first and second robotic machines. The first andsecond robotic machines may coordinate performance of the first sequenceof sub-tasks by the first robotic machine with performance of the secondsequence of sub-tasks by the second robotic machine.

Optionally, the second set of capabilities of the second robotic machinemay include at least one capability that differs from the first set ofcapabilities of the first robotic machine. At least some of thesub-tasks may be sequential such that the second robotic machine may bebegin performance of a corresponding sub-task in the second sequenceresponsive to receiving a notification from the first robotic machinethat the first robotic machine has completed a specific sub-task in thefirst sequence. The first robotic machine may perform the first sequenceof sub-tasks by moving the second robotic machine from a startinglocation to a moved location such that the second robotic machine in themoved location and thereby disposed proximate to the target object ofthe equipment more so than when the second robotic machine was in thestarting location. Responsive to receiving a notification from thesecond robotic machine that at least one of manipulation or inspectionof the target object is complete, the first robotic machine lowers thesecond robotic machine back to the starting location.

In an embodiment, a system includes a first robotic machine that has aset of capabilities for interacting with a surrounding environment. Thefirst robotic machine has a communication circuit that can receive afirst sequence of sub-tasks for performing a task that involves at leastone of manipulating or inspecting a target object of the equipment. Thefirst sequence of sub-tasks may relate to the set of capabilities of thefirst robotic machine. The first robotic machine may perform the firstsequence of sub-tasks. The first robotic machine may communicate with asecond robotic machine during the performance of the first sequence ofsub-tasks. The second robotic machine may perform a second sequence ofsub-tasks for performing the task. Completion of both the first andsecond sequences of sub-tasks accomplishes the task. The first roboticmachine may communicate with the second robotic machine during theperformance of the first sequence of sub-tasks to coordinate with thesecond robotic machine such that the first robotic machine starts acorresponding sub-task in the first sequence responsive to a receivednotification from the second robotic machine that the second roboticmachine has at least one of started or completed a specific sub-task inthe second sequence. Responsive to completing a corresponding sub-taskin the first sequence, the first robotic machine may transmit anotification to the second robotic machine that the correspondingsub-task is complete. The first robotic machine has a movable roboticarm. The set of capabilities include the robotic arm extending relativeto the first robotic machine, grasping the target object, manipulatingthe target object, releasing the target object, and retracting relativeto the first robotic machine.

In one embodiment, each robotic machine is equipped with one or moresensors and tools. Suitable sensors may include force-torque (F/T)sensors, tactile sensors, encoders, cameras, chemical sensors,bio-sensors, lidar, radar, time-of-flight (TOF) sensors, thermometers,pressure sensors, acoustic and vibration sensors, accelerometers (e.g.,position, angle, displacement, speed and acceleration sensors), magneticsensors, electric current or electric potential sensors (e.g., voltagesensors), radiation sensors, and triangulation sensors. Suitabletriangulation sensors may include microwave sensors and camera sensors.In a collaborative robotic team, robotic machines may have differentcapabilities for different tasks or for different subtasks of a givetask. In this way, the sensor function may be distributed across anumber of robotic machines. For example, one robot can have camerasensor, one robot can have an infrared (IR) sensor, and one robot canhave acoustic sensors. The robot with the visual optical camera sensormay have a light source to produce both visible and infrared light. Thecamera might then record an image of a working area on the work object,while the IR sensor may use the IR spectrum from the light source fortriangulation of a tool, being manipulated by its supporting roboticmachine, with regard to the work object. For example, an 880 nm LEDlight source may emit a collimated, near-infrared light beam. The beambounces off the work object and/or another robotic machine, and isreceived by a photodiode positioned adjacent to the LED source. A secondphotodiode (or a linear array of photodiodes) may be positioned fartheralong the length of the sensor. When the emitted beam bounces off adetermine target, the reflected energy is concentrated on the firstadjacent photodiode. When an object, such as an arm with a tool ofanother robotic machine, moves into the optical path, the reflected beambounces back from the object. Because the beam is no longer travelingthe full optical path length, its reflected angle changes. One of theadjacent photodiodes may receive or sense the optical energy and thefirst robotic machine responds by sending a signal to the second roboticmachine. At that point the second robotic machine may slow or stopmovement of the arm to prevent or reduce the chance of a collision. Inthis way, the robotic machines can share information with each other toperform some tasks or subtasks. In one embodiment, various roboticmachines, or portions of such machines, have one or more register marks.These register marks may be sensed by various sensors communicativelycoupled to the task manager or other robotic machines. When sensed, thedistance, orientation, and location of the register mark (and byextension a tool of a robotic machine) may be determined. Additionallyor alternatively, bar codes (2D and/or 3D) disposed on a portion of arobotic machine may be used to both identify the robotic machine (orportion thereof) and act as a register mark when sensed by a sensor.

The robotic machines can have implements (e.g., tools and tool sets)that differ from each other. For example, various robotic machines mayeach have one or more of: 2-finger grippers, multi-finger grippers,magnet grippers, vacuum grippers, screw drivers, wrench, welding tool,rotary saw, grinder, impact hammer, and the like. Accordingly, whileperforming a subtask the sensors from one or more robotic machines maybe used to guide the tools of one or more other robotic machines.

With regard to communication between robotic machines working on a task,in one embodiment a centralized task manager coordinates communicationamong robots and acts as a communication hub. That is, each roboticmachine communicates with the hub but not necessarily with each other.In another embodiment, the robotic machines may have both centralizedcommunication and distributed communication. That is, the roboticmachines can both communicate through the task manager and communicateto each other directly. Alternatively, once a task has been assigned,the robotic machines may only communicate with each other and may notcommunicate back to a central task manager. Further, with regard to anembodiment in which a task manager is not available or is not used, therobotic machines may function autonomously. They may identify tasks tobe done and assign robotic machines to perform that task. They furthermay assign out sub-tasks for that task to the plurality of assignedrobotic machines.

Suitable robotic machines may be mobile and have wheels, tracks, aplurality of legs, rotors, propellers, and the like. Other roboticmachines may be stationary, and the work object and/or other mobilerobotic machines may be brought to the stationary robotic machine. Theconcept of stationary and mobile may be extended to include where anotherwise mobile robotic machine anchors itself, at least temporarily,relative to the work object and/or another robotic machine. In oneembodiment, the robotic machine anchors itself directly to the workobject. It may do this using implements. In another embodiment, therobotic machine anchors itself to a portion of the nearby environment(e.g., the ground). Suitable environmental anchors may includestabilizing legs, drills or augers, clamps, and the like.

In one embodiment, the task manager may include a protected space datasource and an exposed space data source. The protected space data sourcemight store, for each of a plurality of monitoring nodes, a series ofnormal values that represent normal operation of a system such as thosesystems described herein. Such values may be generated by a model orcollected from actual monitoring node data, or simply set as factorystandards. A monitoring node refers to, for example, location signals,sensor data, signals sent to actuators, motors, pumps, and auxiliaryequipment, intermediary parameters that are not direct sensor signalsnot the signals sent to auxiliary equipment, and/or control logical(s).These may represent, for example, monitoring nodes that receive datafrom an exposed monitoring system in a continuous fashion in the form ofcontinuous signals or streams of data or combinations thereof. Thisexposed monitoring system stores data and information in the exposedspace data source. Moreover, the monitoring nodes may be used to monitoroccurrences of communication faults, cyber-threats or other abnormalevents. This data path may be designated specifically with encryptionsor other protection mechanisms so that the information may be securedand not be tampered with via cyber-attacks. The exposed space datasource might store, for each of the monitoring nodes, a series of valuesthat represent an undesirable operation of the system (e.g., when thesystem is experiencing a cyber-attack). Suitable encryption protocolsmay be used, such as hashing (e.g., MD5, RIPEMD-160, RTRO, SHA-1, SHA-2,Tiger, WHIRLPOOL, RNGss, Blum Blum Shub, Yarrowm etc.), key exchangeencryption (e.g., Diffie-Hellman key exchange), symmetric encryptionmethods (e.g., Advanced Encryption Standard (AES), Blowfish, DataEncryption Standard (DES), Twofish, Threefish, IDEA, RC4, TinyEncryption algorithm, etc.), asymmetric encryption methods (e.g.,Rivest-Shamir-Adlemen (RSA), DAS, ElGamal, Elliptic curve cryptography,NTRUEncrypt, etc.), or a combination thereof.

During operation, information from the protected space data source andthe exposed space data source may be evaluated by the task manager toidentify a decision boundary (that is, a boundary that separates desiredbehavior from undesired behavior). If data or information flowing fromthe monitoring nodes, when evaluated, identifies with the protectedspace data source, or within an determined limit relative thereto, thetask manager will continue operation normally. However, if the data orinformation in the exposed space data source crosses the decisionboundary, the task manager may initiate a safe mode in response. Thesafe mode may be, in one embodiment, a soft shutdown mode that itintended to avoid damage or injury based on the shutdown itself.

In one embodiment, the first robotic machine may include one or moresensors to detect one or more characteristics of a target object and asecond robotic machine may include one or more effectors to perform anoperation based on a task assigned by the task manager. The operationmay be to assess, repair, and/or service the target object. The robotsystem may include a processing system that includes one or moreprocessors operatively coupled to memory and storage components. Whilethis may be conceptualized and described in the context of a singleprocessor-based system to simplify explanation, the overall processingsystem used in implementing a task management system as discussed hereinmay be distributed throughout the robotic machines and/or implemented asan off-board centralized control system. With this in mind, theprocessor may generate a set of sub-tasks to assess the target objectfor defects. For example, the task manager may determine a taskincluding the sub-tasks (e.g., desired inspection coverage of the targetobject) and/or resources (e.g., robot machine capabilities) available.Based on the generated set of sub-tasks, the task manager may implementthe task by sending signal(s) to the robotic machines and therebyprovide sub-task instructions to perform the task. A controller of eachrobotic machine may process any received instructions and in turnsignal(s) to one or more implements controlled by the respective roboticmachine to control operation and to perform the assigned sub-tasks.

The task may include a plurality of sub-tasks to be collectivelyperformed by at least the first and second robotic machines. Further,the task manager may adjust (e.g., revise) the sub-tasks based on thedata received from sensors related to the target object. For example,the sub-task may be adjusted based on acquired data indicative of apotential defect of the target object. The task manager may send asignal(s) encoding or conveying instructions to travel a specifieddistance and/or direction that enables the robotic machine to acquireadditional data related to the target object associated with thepotential defect.

Upon performing the assigned tasks, the task manager may assess thequality of data received from the sensors. Due to a variety of factors,the quality of the data may be below a threshold level of quality. Forexample, pressure sensors or acoustic sensors may have background noisedue to the conditions proximate to the target object. As such, the taskmanager may determine a signal-to-noise ratio of the signals from thesensors that indicates a relationship between a desired signal andbackground noise. If the task manager determines that thesignal-to-noise ratio falls below a threshold level of quality, the taskmanager may adapt the sub-task to acquire additional data and/or improvethe quality of the data feed. If the task manager determines that thesignal-to-noise ratio is above a threshold level of quality, the taskmanager may proceed to perform maintenance actions associated with thesub-tasks based on the sensor data.

In certain embodiments, to perform maintenance actions, the task managermay generate, maintain, and update a digital representation of thetarget object based on one or more characteristics that may be monitoredusing robotic machine intermediaries and/or derived from known operatingspecifications. For example, the task manager may create a digitalrepresentation that includes, among other aspects, a 3D structural modelof the target object (which may include separately modeling componentsof the target object as well as the target object as a whole). Such astructural model may include material data for one or more components,lifespan and/or workload data derived from specifications and/or sensordata, and so forth. The digital representation, in some implementationsmay also include operational or functional models of the target object,such as flow models, pressure models, temperature models, acousticmodels, lifting models, and so forth. Further, the digitalrepresentation may incorporate or separately model environmental factorsrelevant to the target object, such as environmental temperature,humidity, pressure (such as in the context of a submersible targetobject, airborne target object, or space-based target object). As partof maintaining and updating the digital representation, one or moredefects in the target object as a whole or components of the targetobject may also be modeled based on sensor data communicated to theprocessing components.

Depending on the characteristics of the structural model, the taskmanager may generate a task specifying one or more tasks or action, suchas acquiring additional data related to the target object. For example,if the task manager determines that acquired data of a location on thestructural model is below a threshold quality level or is otherwiseinsufficient, the task manager may generate or update a revised taskthat includes one or more tasks that position the robot to acquireadditional data regarding the location.

Sensor data may be used to generate, maintain, and update the digitalrepresentation, including modeling of defects. The sensors used tocollect the sensor data may vary between robotic machines. Example ofsensors include, but are not limited to, cameras or visual sensorscapable of imaging in one or more of visible, low-light, ultraviolet,and or infrared (i.e., thermal) contexts, thermistors or othertemperature sensors, material and electrical sensors, pressure sensors,acoustic sensors, radiation sensors or imagers, probes that applynon-destructive testing technology, and so forth. With respect toprobes, for example, the robotic machine may contact or interactphysically with the target object to acquire data.

The environment's digital representation may incorporate or be updatedbased on a combination of factors derived from the data of one or moresensors on the robotic machine (or integral to the target objectitself). Acquired sensor data may be subjected to a feature extractionalgorithm. In some implementations, relatively faster processing can beachieved by performing feature extraction on data obtained by RGB orinfrared cameras. In some implementations, scale-invariant featuretransform (SIFT) and speeded up robust features (SURF) techniques mayprovide additional information on descriptors—for example, Oriented FASTand rotated BRIEF (ORB) feature extraction can be performed. In otherimplementations, simple color, edge, corner, and plane features can beextracted.

The task manager may receive visual image data from imaging sensors(e.g., cameras, lidar) on the robotic machines to create or update a 3Dmodel of the target object to localize defects on the 3D model. Based onthe sensor data, as incorporated into the 3D model, the task manager maydetect a defect, such as a crack, a region of corrosion, or missingpart, of the target object. For example, the task manager may detect acrack on a location of a vehicle based on visual image data thatincludes color and/or depth information indicative of the crack. The 3Dmodel may additionally be used as a basis for modeling other layers ofinformation related to the target object. Further, the task manager maydetermine risk associated with a potential or imminent defect based onthe digital representation. Depending on the risk and a severity of thedefect, the task manager, as described above, may send signal(s) to therobotic machines indicating instructions to repair or otherwise addressa present or pending defect.

In some embodiments, to repair, remediate, or otherwise prevent adefect, the task manager may create a 3D model of a part or componentpieces of the part of the target object needed for the repair. The taskmanager may generate descriptions of printable parts or part components(i.e., parts suitable for generation using additive manufacturingtechniques) that may be used by a 3D printer (or other additivemanufacturing apparatus) to generate the part or part components. Basedon the generated instructions or descriptions, the 3D printer may createthe 3D printed part to be attached to or integrated with the targetobject as part of a repair process. Further, one or more roboticmachines may be used to repair the target object with the 3D printedpart(s). While a 3D printed part is described in this example, otherrepair or remediation approaches may also be employed. For example, inother embodiments, the task manager may send signal(s) indicatinginstructions to a controller of a robotic machine to control the roboticmachine to spray a part of the target object (e.g., with a lubricant orspray paint) or to replace a part of the target object from an availableinventory of parts. Similarly, in some embodiments, a robotic machinemay include a welding apparatus that may be autonomously employed toperform an instructed repair. In some embodiments, the task manager maysend signal(s) to a display to indicate to an operator to enable theoperator to repair the defect using one or more implements of therobotic machines.

Accurate position control and navigation of an unmanned aerial vehicle(UAV) in GPS denied environments, such as indoors, is challenging asalternatives are to rely on inertial sensors or other sources such asvision. Inertial sensing is prone to drifts and biases, whilealternative techniques such as vision-based are computationallyintensive. This invention proposes a system of one or more laser beamtransmitters, one or more reflectors, and quad detectors (or cameras)mounted on gimbals on the drone. The laser transmitter through thereflector transmits a laser beam in any desired direction. For anydesired height of the drone, exact position in space could be specifiedusing the spherical angles at which the laser beam is transmitted. Thedrone, upon achieving the way-point is then stabilized by a controllerthat locks the laser beam to the quad detector. Due to positioning ornavigational errors, if the drone does not achieve detection of thelaser beam in the expected amount of time, the laser beam is rasterscanned till the quad detector achieves detection. Upon the initiallaser lock, the laser beam angle is shifted within specified maximumangular rate such that given commanded height, the drone can navigatewhile maintaining laser lock. This is achieved by a controller thattries to minimize the offset of laser beam impact point on the detectorand the detector center or a pre-specified radius around the detectorcenter. If laser lock is lost during guidance, then either the laserbeam could be moved to the previous position where lock was achieved,while keeping the drone in hover mode or the drone could navigate basedon inertial sensing to the current waypoint specified by laser beamangle and commanded height, or a combination of the two. In order toachieve this feedback, quad detector measurements could be provided tothe laser transmitter/mirror controller.

With regard to robotic machine positioning and stabilization, the poseestimate of the robotic machine camera can be used to localize an objector region within the field of view of the camera through trigonometrictransforms. This setup can be used to localize artifacts on, forexample, walls, trees, buildings, etc. during inspection for offlineanalysis or for providing the next waypoint for the robotic machine tomake a closer inspection. In one embodiment, a laser transmittingunit(s) mounted on a first robotic machine interacts with a detectormounted on the second robotic machine. The laser transmitter includes alaser source, which may be incident on a mirror mounted on amotor-controlled stage. Motors may be used to actuate the mirror, and togive pan and tilt flexibility. The flexibility allows for the steeringor pointing of the laser. A detector mounted on the second roboticmachine may have one or more detection units. The detection unit mayinclude a focusing lens to better direct the laser beam and increase afield of view. The second robotic machine may start its travel, e.g.,flight, from a position where the detector sees the incident laserlight. With an initial lock of the detector onto the laser light, themotors connected to the mirror actuate as required to steer the lasertoward a region of interest. Using this knowledge and applying geometrictransformations, the pan and tilt angles may be converted to positionalfeedback to the second robotic machine, where the robotic machine'scontrol unit then reacts accordingly such that the laser beam is kept atthe center of the detector. Height and/or distance information may becollected from an altitude measurement sensor (affixed to the secondrobotic machine). Such z-dimensional information may aid in 3D movement.

In one embodiment, multiple lasers and detectors may be used to increasethe accuracy of the system and navigation. The second robotic machinemay be controlled by the first robotic machine even if an obstacleoccludes individual beams from reaching the detector. For situationswhere there is a requirement to see around corners, multiple roboticmachines can be deployed, and an intermediate robotic machine may act asa repeater. The intermediate robotic machine may be deployed with a beamsplitter+detector installed. The beam splitter may reflect half of theincident light on to the second robotic machine and its detector. Theremaining half of the light may reach a detector present on theintermediate robotic machine. Using the same laser-locking strategy, theposition of the intermediate robotic machine may be known.

The imaging sensor of the second robotic machine may be used for bothnavigation/localization as well as for inspection purposes. Further, thelaser may be used for both navigation/localization as well as forline-of-sight data transmission.

Although embodiments of the subject matter are described herein withrespect to mobile equipment and vehicles, embodiments of the inventivesubject matter are also applicable for use with other equipmentgenerally. Suitable vehicles may include rail cars, trains, locomotives,and other rail vehicles. Other suitable vehicles may include miningvehicles, agricultural vehicles, and other off-highway vehicles (e.g.,vehicles that are not designed or permitted to travel on publicroadways), automotive or passenger vehicles, aircraft (manned andunmanned), marine vessels, and/or freight transportation vehicles (e.g.,semi-tractor/trailers and over-the-road trucks). The term ‘consist’refers to two or more robotic machines or items of mobile equipment thatare mechanically or logically coupled to each other. By logicallycoupled, the plural items of mobile equipment are controlled so thatcontrols to move one of the items causes a responsive movement (e.g., acorresponding movement) in the other items in consist, such as bywireless command.

In an embodiment, a system includes a first robotic machine having afirst set of capabilities for interacting with a target object, a secondrobotic machine having a second set of capabilities for interacting withthe target object, and a task manager. The task manager has one or moreprocessors and is configured to determine capability requirements toperform a task on the target object; the task has an associated seriesof sub-tasks, with the sub-tasks having one or more capabilityrequirements. The task manager is also configured to assign a firstsequence of sub-tasks to the first robotic machine for performance bythe first robotic machine based at least in part on the first set ofcapabilities and a second sequence of sub-tasks to the second roboticmachine for performance by the second robotic machine based at least inpart on the second set of capabilities. The first and second roboticmachines are configured to coordinate performance of the first sequenceof sub-tasks by the first robotic machine with performance of the secondsequence of sub-tasks by the second robotic machine, and thereby toaccomplish the task.

In another embodiment, the first and second sets of capabilities of thefirst and second robotic machines each include at least one of flying,driving, diving, lifting, imaging, grasping, rotating, tilting,extending, retracting, pushing, and/or pulling. In another embodiment,the second set of capabilities of the second robotic machine include atleast one capability that differs from the first set of capabilities ofthe first robotic machine. In another embodiment, the first and secondrobotic machines coordinate performance of the first sequence ofsub-tasks by the first robotic machine with the performance of thesecond sequence of sub-tasks by the second robotic machine bycommunicating directly with each other. In another embodiment, the firstrobotic machine notifies the second robotic machine, directly orindirectly, that one of the corresponding sub-tasks is complete and thesecond robotic machine is responsive to the notification by performing acorresponding sub-task in the second sequence.

In another embodiment, the first robotic machine provides to the secondrobotic machine, directly or indirectly, a sensor signal havinginformation about the target object, and the task manager makes adecision whether the second robotic machine proceeds with a sub-task ofthe second sequence based at least in part on the sensor signal. Inanother embodiment, at least some of the sub-tasks are sequential suchthat the second robotic machine begins performance of a dependentsub-task in the second sequence responsive to receiving a notificationfrom the first robotic machine that the first robotic machine hascompleted a specific sub-task in the first sequence. In anotherembodiment, the first robotic machine performs at least one of thesub-tasks in the first sequence concurrently with performance of atleast one of the sub-tasks in the second sequence by the second roboticmachine.

In another embodiment, the task manager is configured to access adatabase that stores capability descriptions corresponding to eachrobotic machine in a group of robotic machines, and the task manager isfurther configured to select the first and second robotic machines toperform the task instead of other robotic machines in the group based ona suitability of the capability descriptions of the first and secondrobotic machines relative to capability needs ascribed in the databaseto the task or corresponding sub-tasks.

In another embodiment, the first robotic machine performs one or more ofthe first sequence of sub-tasks by coupling to and lifting the secondrobotic machine from a starting location to a lifted location such thatthe second robotic machine in the lifted location is positioned relativeto the target object to complete one or more of the second sequence ofsub-tasks than if the second robotic machine is in the startinglocation. In another embodiment, the first robotic machine performs thefirst sequence of sub-tasks by flying, and the first robotic machineidentifies the target object and determines at least two of: a positionof the target object, a position of the first robotic machine, and aposition of the second robotic machine, and the second robotic machineperforms the second sequence of sub-tasks by one or more of modifyingthe target object, manipulating the target object, observing the targetobject, interacting with the target object, and releasing the targetobject.

In another embodiment, the first robotic machine, having been assigned asequence of sub-tasks by the task manager: determines to travel adetermined path from a first location to a second location, ordetermines to act using a capability of the first set of capabilities,or both determines to travel the intended path and determines to actusing the capability, and signals to the second robotic machine, to thetask manager, or both the second robotic machine and the task managerinformation including at least one of the determined path, the act ofusing the capability, or both. In another embodiment, the second roboticmachine, responsive to the signal from the first robotic machine,initiates a confirmatory receipt signal back to the first roboticmachine.

In another embodiment, the first robotic machine and the second roboticmachine each are configured to generate one or more of: time indexingsignals associated one or both of the first sequence of sub-tasks andthe second sequence of sub-tasks, position indexing signals forlocations of one or both of the first robotic machine and the secondrobotic machine, and orientation indexing signals for one or more toolsconfigured to implement one or both of the first set of capabilities ofthe first robotic machine and the second set of capabilities of thesecond robotic machine.

In another embodiment, at least one of the first robotic machine and/orthe second robotic machine has a first mode of operation that is a fast,gross movement mode and a second mode of operation that is a slow, finemovement mode.

In another embodiment, the system further includes one or more of astabilizer, an outrigger, and/or a clamp, and a transition in operationfrom the first mode to the second mode comprises deploying and settingthe stabilizer, outrigger, or clamp.

In another embodiment, the first mode of operation includes moving atleast one of the first robotic machine and the second robotic machine toa determined location relative to the target object. The second mode ofoperation includes actuating one or more tools of at least one of thefirst robotic machine and the second robotic machine accomplish the taskor a sub-task.

In an embodiment, a system includes a first robotic machine and a secondrobotic machine. The first robotic machine has a first set ofcapabilities for interacting with a surrounding environment. The firstrobotic machine is configured to receive a first sequence of sub-tasksrelated to the first set of capabilities of the first robotic machine.The second robotic machine has a second set of capabilities forinteracting with the surrounding environment. The second robotic machineis configured to receive a second sequence of sub-tasks related to thesecond set of capabilities of the second robotic machine. The first andsecond robotic machines are configured to perform the first and secondsequences of sub-tasks, respectively, to accomplish a task that involvesat least one of manipulating or inspecting a target object that isdistinct from the first and second robotic machines. The first andsecond robotic machines are configured to coordinate performance of thefirst sequence of sub-tasks by the first robotic machine withperformance of the second sequence of sub-tasks by the second roboticmachine.

In another embodiment, at least some of the sub-tasks are sequentialsuch that the second robotic machine begins performance of acorresponding sub-task in the second sequence responsive to receiving anotification from the first robotic machine that the first roboticmachine has completed a specific sub-task in the first sequence. Anotherembodiment relates to a method for controlling a first robotic machineand a second robotic machine. The first robotic machine has a first setof capabilities for interacting with a surrounding environment. Thefirst robotic machine is configured to receive a first sequence ofsub-tasks related to the first set of capabilities of the first roboticmachine. The second robotic machine has a second set of capabilities forinteracting with the surrounding environment. The second robotic machineis configured to receive a second sequence of sub-tasks related to thesecond set of capabilities of the second robotic machine. The methodincludes performing the first and second sequences of sub-tasks toaccomplish a task comprising at least one of manipulating or inspectinga target object, and coordinating performance of the first sequence ofsub-tasks by the first robotic machine with performance of the secondsequence of sub-tasks by the second robotic machine.

The above description is illustrative and not restrictive. For example,the above-described embodiments (and/or aspects thereof) may be used incombination with each other. In addition, modifications may be made toadapt a situation or material to the teachings of the inventive subjectmatter without departing from its scope. While the dimensions and typesof materials described herein are intended to define the parameters ofthe inventive subject matter, they are not limiting and are exampleembodiments. Many other embodiments will be apparent to those ofordinary skill in the art upon reviewing the above description. Thescope of the inventive subject matter should, therefore, be determinedwith reference to the appended claims, along with the full scope ofequivalents to which such claims are entitled. In the appended claims,the terms “including” and “in which” are used as the plain-Englishequivalents of the respective terms “comprising” and “wherein.”Moreover, in the following claims, the terms “first,” “second,” and“third,” etc. are used merely as labels, and are not intended to imposenumerical requirements on their objects. Further, the limitations of thefollowing claims are not written in means-plus-function format and arenot intended to be interpreted based on 35 U.S.C. § 112(f), unless anduntil such claim limitations expressly use the phrase “means for”followed by a statement of function void of further structure.

The foregoing description of certain embodiments of the inventivesubject matter will be better understood when read in conjunction withthe appended drawings. To the extent that the figures illustratediagrams of the functional blocks of various embodiments, the functionalblocks are not necessarily indicative of the division between hardwarecircuitry. Thus, for example, one or more of the functional blocks (forexample, processors or memories) may be implemented in a single piece ofhardware (for example, a general-purpose signal processor,microcontroller, random access memory, hard disk, and the like).Similarly, the programs may be stand-alone programs, may be incorporatedas subroutines in an operating system, may be functions in an installedsoftware package, and the like. The various embodiments are not limitedto the arrangements and instrumentality shown in the drawings. As usedherein, an element or step recited in the singular and proceeded withthe word “a” or “an” should be understood as not excluding plural ofsaid elements or steps, unless such exclusion is explicitly stated.Furthermore, references to “one embodiment” of the inventive subjectmatter are not intended to be interpreted as excluding the existence ofadditional embodiments that also incorporate the recited features.Moreover, unless explicitly stated to the contrary, embodiments“comprising,” “including,” or “having” an element or a plurality ofelements having a particular property may include additional suchelements not having that property.

This written description uses examples to disclose several embodimentsof the inventive subject matter and also to enable a person of ordinaryskill in the art to practice the embodiments of the inventive subjectmatter, including making and using any devices or systems and performingany incorporated methods. The patentable scope of the inventive subjectmatter is defined by the claims, and may include other examples thatoccur to those of ordinary skill in the art. Such other examples areintended to be within the scope of the claims if they have structuralelements that do not differ from the literal language of the claims, orif they include equivalent structural elements with insubstantialdifferences from the literal languages of the claims.

What is claimed is:
 1. A system, comprising: a first robotic machinehaving a first set of capabilities for interacting with a target objecton stationary equipment; a second robotic machine having a second set ofcapabilities for interacting with the target object, at least one of thefirst robotic machine or the second robotic machine having a first modeof operation that is a fast, gross movement mode and a second mode ofoperation that is a slow, fine movement mode; and a task manager havingone or more processors and that is configured to determine capabilityrequirements to perform a task on the target object, the task having anassociated series of sub-tasks, with the sub-tasks having one or morecapability requirements, the task manager being configured to assign afirst sequence of sub-tasks within the associated series of sub-tasks tothe first robotic machine for performance by the first robotic machinebased at least in part on the first set of capabilities, and to assign asecond sequence of sub-tasks within the associated series of sub-tasksto the second robotic machine for performance by the second roboticmachine based at least in part on the second set of capabilities, thefirst and second robotic machines being configured to coordinateperformance of the first sequence of sub-tasks by the first roboticmachine with performance of the second sequence of sub-tasks by thesecond robotic machine, and thereby to accomplish the task.
 2. Thesystem of claim 1, wherein the stationary equipment is a railroad track,a road, a bridge, a building, a stack, or a stationary machine.
 3. Thesystem of claim 1, wherein the first and second sets of capabilities ofthe first and second robotic machines each include at least one offlying, driving, diving, lifting, imaging, grasping, rotating, tilting,extending, retracting, pushing, or pulling.
 4. The system of claim 1,wherein the second set of capabilities of the second robotic machineincludes at least one capability that differs from the first set ofcapabilities of the first robotic machine.
 5. The system of claim 1,wherein the first and second robotic machines coordinate performance ofthe first sequence of sub-tasks by the first robotic machine with theperformance of the second sequence of sub-tasks by the second roboticmachine by communicating directly with each other.
 6. The system ofclaim 1, wherein the first robotic machine notifies the second roboticmachine, directly or indirectly, that one of the corresponding sub-tasksis complete, and the second robotic machine is configured to perform acorresponding sub-task in the second sequence responsive to beingnotified.
 7. The system of claim 1, wherein the first robotic machineprovides to the second robotic machine, directly or indirectly, a sensorsignal having information about the target object, and the task manageris configured to decide whether the second robotic machine proceeds witha sub-task of the second sequence based at least in part on the sensorsignal.
 8. The system of claim 1, wherein at least some of the sub-tasksare sequential such that the second robotic machine begins performanceof a dependent sub-task in the second sequence responsive to receiving anotification from the first robotic machine that the first roboticmachine has completed a specific sub-task in the first sequence.
 9. Thesystem of claim 1, wherein the first robotic machine concurrentlyperforms at least one of the sub-tasks in the first sequence withperformance of at least one of the sub-tasks in the second sequence bythe second robotic machine.
 10. The system of claim 1, wherein the taskmanager is configured to access a database that stores capabilitydescriptions corresponding to each robotic machine in a group of roboticmachines, and the task manager is further configured to select the firstand second robotic machines to perform the task instead of other roboticmachines in the group based on a suitability of the capabilitydescriptions of the first and second robotic machines relative tocapability needs ascribed in the database to the task or correspondingsub-tasks.
 11. The system of claim 1, wherein the first robotic machineis configured to perform one or more sub-tasks of the first sequence ofsub-tasks by moving the second robotic machine from a first location toa second location such that the second robotic machine in the secondlocation is positioned relative to the target object to complete one ormore sub-tasks of the second sequence of sub-tasks.
 12. The system ofclaim 11, wherein the first robotic machine is configured to identifythe target object and determine at least two of: a position of thetarget object, a position of the first robotic machine, or a position ofthe second robotic machine, and the second robotic machine is configuredto perform one or more sub-tasks of the second sequence of sub-tasks bymanipulating the target object.
 13. The system of claim 1, wherein thefirst robotic machine, having been assigned a sequence of sub-tasks bythe task manager: determines to travel a determined path from a firstlocation to a second location, or determines to act using a capabilityof the first set of capabilities, or both determines to travel thedetermined path and determines to act using the capability, and signalsto the second robotic machine, to the task manager, or both the secondrobotic machine and the task manager, information including at least oneof the determined path, the act of using the capability, or both. 14.The system of claim 1, wherein the first robotic machine and the secondrobotic machine each are configured to generate one or more of: timeindexing signals associated with one or both of the first sequence ofsub-tasks or the second sequence of sub-tasks, position indexing signalsfor locations of one or both of the first robotic machine or the secondrobotic machine, or orientation indexing signals for one or more toolsconfigured to implement one or both of the first set of capabilities ofthe first robotic machine or the second set of capabilities of thesecond robotic machine.
 15. The system of claim 1, further comprisingone or more of a stabilizer, an outrigger, or a clamp, and wherein atransition in operation from the first mode to the second mode comprisesdeploying and setting said one or more of the stabilizer, the outrigger,or the clamp.
 16. The system of claim 1, wherein the first mode ofoperation comprises moving at least one of the first robotic machine orthe second robotic machine to a determined location relative to thetarget object; and the second mode of operation comprises actuating oneor more tools of at least one of the first robotic machine or the secondrobotic machine to accomplish the task or a sub-task.
 17. A system,comprising: a first robotic machine having a first set of capabilitiesfor interacting with stationary equipment, the first robotic machinebeing configured to receive a first sequence of sub-tasks related to thefirst set of capabilities of the first robotic machine; and a secondrobotic machine having a second set of capabilities for interacting withthe stationary equipment, the second robotic machine being configured toreceive a second sequence of sub-tasks related to the second set ofcapabilities of the second robotic machine, at least one of the firstrobotic machine or the second robotic machine having a first mode ofoperation that is a fast, gross movement mode and a second mode ofoperation that is a slow, fine movement mode, and the first and secondrobotic machines are configured to coordinate performance of the firstand second sequences of sub-tasks, respectively, to accomplish a taskthat involves manipulating a target object that is distinct from thefirst and second robotic machines, the target object being on thestationary equipment, and the first robotic machine is configured toprovide to the second robotic machine, directly or indirectly, a sensorsignal having information about the target object, and the first roboticmachine is configured to perform one or more sub-tasks of the firstsequence of sub-tasks by identifying the target object and moving thesecond robotic machine from a first location to a second location suchthat the second robotic machine at the second location is positionedrelative to the target object to complete one or more sub-tasks of thesecond sequence of sub-tasks, the second robotic machine being closer tothe target object at the second location than at the first location, andthe second robotic machine is configured to perform one or moresub-tasks of the second sequence of sub-tasks at the second location bymanipulating the target object based at least in part on the sensorsignal.
 18. The system of claim 17, wherein at least some of thesub-tasks are sequential such that the second robotic machine beginsperformance of a dependent sub-task in the second sequence responsive toreceiving a notification from the first robotic machine that the firstrobotic machine has completed a specific sub-task in the first sequence.19. A method for a first robotic machine having a first set ofcapabilities for interacting with stationary equipment and a secondrobotic machine having a second set of capabilities for interacting withthe stationary equipment, the first robotic machine being configured toreceive a first sequence of sub-tasks related to the first set ofcapabilities of the first robotic machine, the second robotic machinebeing configured to receive a second sequence of sub-tasks related tothe second set of capabilities of the second robotic machine, at leastone of the first robotic machine or the second robotic machine having afirst mode of operation that is a fast, gross movement mode and a secondmode of operation that is a slow, fine movement mode, the methodcomprising: coordinating performance of the first sequence of sub-tasksby the first robotic machine with performance of the second sequence ofsub-tasks by the second robotic machine; and performing the first andsecond sequences of sub-tasks to accomplish a task comprisingmanipulating a target object on the stationary equipment; wherein thefirst robotic machine performs one or more sub-tasks of the firstsequence of sub-tasks by: identifying the target object on thestationary equipment; determining at least two of: a position of thetarget object, a position of the first robotic machine, and a positionof the second robotic machine; and providing to the second roboticmachine, directly or indirectly, a sensor signal having informationabout the target object, and the second robotic machine performs one ormore sub-tasks of the second sequence of sub-tasks by manipulating thetarget object based on the sensor signal.